Repository navigation
[PR] feat: host message ids into the canonical store — reconcile becomes exact set algebra (#3464 R1) - #3473
ranxianglei wants to merge 3 commits into
Conversation
📦 Built Package ArtifactBranch: Option A — Install this exact build (recommended)npm install -g billion-context@0.0.4116-pr.3473.e7a646aThis exact version is immutable and reproducible. The rolling Option B — Download artifact
tar xzf billion-context-pr3473.tgz
npm install -g packageInstall instructions are refreshed on each push. |
[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b) 查重: 不重复。 全仓检索(open+closed)「host message id / _bili_message_ids / 正史收编」仅命中本 issue 与其母 issue #3464。相关项只有 PR #2981(closed unmerged,#2938 Tier-1 tool-call-id 通道半边)——已被本 PR 的 per-message 设计取代,master 上无同类在途实现,无并行工作冲突。 开始 review:已拉取分支 |
(ai: vllm-qwen/qwen3.8-27b) Review 结论:暂不可合(拦下)。 方向正确、降级路径干净、无回归,但核心配对函数把「wire 对齐的 id 列表」和「展开后的 core 数组」当成同一个东西在按位置 zip——实跑证明:任何含多块消息(text+tool_use、text+tool_result,即任何带工具调用的回合)的会话里,整个 host-id 通道静默失效。验收标准 #1/#3 在真实流量下不成立。另有 1 处 CI 红已顺手修掉推上分支,1 处前提需要补端到端证据。 F1(拦下):coreHostRefs 的配对契约与生产数据形状不符生产链路永远是这两个形状:
而 后果(全部下游都挂在同一个函数上):
测试把这个错误契约钉成了绿的: 修法方向(给作者,几行级但牵动测试面): F2(已直接修,commit 472fb02)CI F3(需补证据):「宿主真的把 id 序列化进出站 body」是承重前提,仓内零证据
F4(合并前注意)分支落后 master 5 个 commit(fa41483),其中三个动过 本地定向验证(PR head 2ff663d)新套件 27/27 过;触碰的既有套件(fold-reconcile/storage-ext/storage-ingest-watermark/storage-unified)99 过 1 skip 0 败;typecheck、build 干净。全量回归以远程 CI 为准(已在 472fb02 上重跑)。 八问
判定:拦下,待 F1 修复(含 fixture 改写)+ F3 端到端证据 + F4 rebase 后复审。 F2 已落地(472fb02,CI 重跑中)。 |
… capability Review fix for #3473 (separate from the feature commit): the feature adds capabilities.hostMessageIds to the plugin manifest but did not regenerate the golden snapshots, so the wire-contract-golden gate fails on CI (#1304: ledger only grows; goldens change only via explicit regeneration). Justification for the schema change: additive capability advertisement for the #3464 R1 host message-id side channel — old plugins ignore the unknown capability and never stamp; the field name and cap are the channel contract plugins learn at bootstrap. Regenerated with node --import tsx scripts/update-wire-contract-goldens.ts; the only delta is the hostMessageIds block in both manifest snapshots.
(ai: vllm-qwen/qwen3.8-27b) 补一条 review 发现 + 已直接修(commit b143d69): 上一条回复后我核对了新 head 的全量 CI, 至此我在本分支上共推了 2 个独立标注的 review 修复 commit: |
b143d69 to
2ef562a
Compare
… capability Review fix for #3473 (separate from the feature commit): the feature adds capabilities.hostMessageIds to the plugin manifest but did not regenerate the golden snapshots, so the wire-contract-golden gate fails on CI (#1304: ledger only grows; goldens change only via explicit regeneration). Justification for the schema change: additive capability advertisement for the #3464 R1 host message-id side channel — old plugins ignore the unknown capability and never stamp; the field name and cap are the channel contract plugins learn at bootstrap. Regenerated with node --import tsx scripts/update-wire-contract-goldens.ts; the only delta is the hostMessageIds block in both manifest snapshots.
…comes exact set algebra (#3464 R1) Plugin lanes (dsh/opencode) passively read each wire message's own host id off the outgoing body and stamp it in the reserved top-level field _bili_message_ids when the proxy advertises capabilities.hostMessageIds. The proxy extracts+strips the field at the body-parse chokepoint, zips the wire-aligned list onto cores via wireIndex sidecars, and persists it: - schema v6: messages gains host_id/host_slot + partial unique index (session_id, host_id, host_slot); migration additive/probe-guarded - planIngest dedups stamped legs by host key — one row per host message across drifted resends (fresher bytes win, first-seen position kept); unstamped lanes keep today's byte-keyed rows bit-for-bit - store reuses the host-keyed seq across byte drift and refreshes the row in place (documented append-only exception for #2960(5) digest parity) - fold-reconcile Pass -1 claims missing covered ids by exact host-id equality (role-gated, k-th within multi-core groups) ahead of every byte-based pass; anchor.h seeds/backfills across passes; unstamped lanes are bit-identical - fork snapshots expose an additive hostIds array mapping each ordered message back to its bili ref (outside every hash) No new config/env surface. No kernel changes.
… capability Review fix for #3473 (separate from the feature commit): the feature adds capabilities.hostMessageIds to the plugin manifest but did not regenerate the golden snapshots, so the wire-contract-golden gate fails on CI (#1304: ledger only grows; goldens change only via explicit regeneration). Justification for the schema change: additive capability advertisement for the #3464 R1 host message-id side channel — old plugins ignore the unknown capability and never stamp; the field name and cap are the channel contract plugins learn at bootstrap. Regenerated with node --import tsx scripts/update-wire-contract-goldens.ts; the only delta is the hostMessageIds block in both manifest snapshots.
2ef562a to
a94d371
Compare
(ai: vllm-qwen/qwen3.8-27b) 冲突已解决,rebase 完成。 新 head: 冲突实况:全分支只有 1 处内容冲突,在 定向验证(本地只跑改动相关范围,全量以远程 CI 为准):typecheck 干净; 八问
收工判定:冲突已清除,分支骑在最新 master 上,CI 在新 head 上重跑中。注意:这不改变 review 结论——F1( |
|
Superseded as an implementation: written against the pre-#3642/#3658 identity world (whole-store re-key assumptions, pre-snapshot-render reconcile). The R1 direction — host-declared message ids into the canonical store, exact set-algebra reconcile — remains tracked by #3464, and the current architecture (canonical seeds, media fingerprints, snapshot render seams) is the base it should be rebuilt on. Closing the branch; reopen as a fresh PR against master when R1 is scheduled. |
(ai: vllm-qwen/qwen3.8-27b)
Closes #3464
What this does (R1 of the #2960 identity pipeline)
Hosts that serialize stable per-message ids (dsh
Message.id, opencodemsg_…) now ride those ids through bili end-to-end. On stamped lanes, message identity stops being byte-hash guessing and becomes exact set algebra:idoff the outgoing body and stamp it in the reserved top-level field_bili_message_ids— only when the proxy advertisescapabilities.hostMessageIdsin its manifest. The proxy extracts + unconditionally strips the field at the single body-parse chokepoint (src/server/handle.ts), so the reserved namespace can never reach upstream.messages.host_id/host_slotplus partial unique index(session_id, host_id, host_slot) WHERE host_id IS NOT NULL.planIngestdedups stamped legs by host key: one row per host message across drifted resends (fresher bytes win, first-seen position kept). Store reuses the host-keyed seq across byte drift and refreshes the row in place.anchor.hseeds and backfills across passes so a later unstamped request degrades gracefully to the heuristic ladder.hostIdsarray mapping each ordered message back to its bili ref (outside every hash by construction).Lanes without host ids (pi/omp message model has no id field per the #2938 survey; claude/codex/kimi/zcode) keep today's byte-keyed behavior bit-for-bit — gradual adoption, not protocol-breaking. No kernel changes, no new config/env surface.
Behavior changes (old → new)
_bili_message_ids: old = unknown top-level field could reach upstream via re-serialization (accidental leak surface); new = extracted + stripped pre-forward on every path. All other bytes unchanged.byHostcounter added to every result literal): active ONLY when the request carries valid host ids AND anchors recorded them; unstamped lanes produce bit-identical plans/counters (pinned by test).capabilities.hostMessageIds; old proxies never see the field, old plugins against new proxies simply don't stamp.hostIdsarray (old clients ignore unknown fields; matching/hash logic untouched).Scope interpretation note
Issue acceptance says "dsh(pi 亦然)", but pi's message model carries no per-message id (#2938 lane survey) — so the mechanism is implemented generic-per-lane with adoption following host serialization capability: dsh + opencode get stamps, pi/omp deliberately NOT wired (a reader that can never fire is dead code). The passive read lights up automatically for any future host (or host build) that starts serializing message ids into the anthropic body. Active injection (forcing ids into the body via waterfall/transform hooks) is a follow-up pending evidence of which hosts serialize them — the passive path makes that optional.
Verification
npm run typecheckclean;npm run buildclean (zero kernel changes — nokernel-changemarker needed).tests/issue3464-host-ids.test.ts(27 tests): channel extract/validate/strip matrix, slot semantics, schema v5→v6 migration + fresh DB, planIngest twin elimination / no-id control / slot disambiguation / fork ref mapping, store seq continuity + in-place refresh, Pass -1 claim/role-guard/k-th/unstamped-inert/anchor-persistence.byHost: 0added to one deep-equal literal intests/fold-reconcile.test.ts;user_versionassertions 5→6 in three storage tests (the watermark test's synthesized DDL now strips the v6 additions too, so it exercises the real v4→v6 path).proxy-persist.test.tsscheduleSave/flushSync entry-count assertions (environmental temp-dir behavior); full-suite verdict = remote CI.Deferred (tracked, not silent)
sinceSeq+ host-id continuity check) — benefit listed in the issue, acceptance doesn't require it; hook point already noted atsrc/incremental-lane.ts(its header tracks "explicit edit/branch ops (R1 message ids)" on Delta lane: native-mode clients send only deltas, server maintains full state + one-time fold, upstream receives snapsho | 增量车道:native 模式客户端只发增量,服务端全量维护+一次折叠,上游只收快照(#2926 的泛化) #3248).