Skip to content

[PR] fix(incremental): continue repeated-message identities across delta turns - #5743

Closed
mariopasquali wants to merge 0 commit into
ranxianglei:masterfrom
mariopasquali:2026-11-05_incremental-content-clusters
Closed

mariopasquali wants to merge 0 commit into
ranxianglei:masterfrom
mariopasquali:2026-11-05_incremental-content-clusters

Conversation

@mariopasquali

Copy link
Copy Markdown
Contributor

When each delta is converted independently, the content-identity cluster counter restarts. Repeating a prompt such as continue reuses an earlier canonical ID, and a cache mark for the new message can also render on its older twin.

Continue the derived-ID cluster suffixes from the acknowledged canonical prefix and move cache marks to the resulting IDs. Use the highest retained suffix so a sparse restored prefix cannot collide with a retained occurrence. Resync still starts a fresh conversion. Client-pinned IDs retain precedence through the existing validation and pinning path; the kernel's ID generator is unchanged.

The six-turn repeated-prompt regression fails on the preceding compatibility patch and passes with this change. Differential tests compare IDs and cache marks with full conversion across every split of a 40-message history. Additional cases cover sparse prefixes, preserved payload objects and continuation followed by client pinning. Typecheck, 15 targeted test files (runner reports 144 passed, 0 failed), build and distribution checks passed on commit 9c3faf3cb9d3b7ab2b6d498892c47125cd552f87.

This is stacked on #5742, based on commit a6ea2a0af152c3ea7270e95dbb5e8db4d2f06e60. It does not repair historical migration identity conflicts. No full repository CI or fresh live OpenCode end-to-end result is claimed by these targeted loopback tests.

Fixes #5741

@ranxianglei ranxianglei changed the title fix(incremental): continue repeated-message identities across delta turns [PR] fix(incremental): continue repeated-message identities across delta turns Nov 5, 2026
@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

收到,开始审核这个 PR。先给接单结论:

查重: 不重复。 open/closed 全量检索过(关键词: repeated content / canonical ID / cluster counter / cache mark twin):没有其它线程覆盖「增量车道每次 delta 独立转换导致 content-ID 簇计数器归零、重复消息复用旧行身份」这一症状+机制组合。最接近的 #5610(整代缓存命中塌到 1–12%)是不同症状不同机制;#5569(omp advisor 串扰)是会话键控问题,不相关。本线程也没有并列的第二个修复 PR(唯一在途的就是本 PR;它叠在 #5742 之上,#5742 是独立的兼容补丁线程)。

标签: bug / opencode / scope/wiring / p2(缺陷类;回归证据来自 OpenCode V2 车道的滚动缓存标记场景,但修的是共享增量车道;协议接线层而非内核——kernel ID 生成器未动;p2:特定条件触发(增量车道 + 跨回合字节相同的重复消息),后果是缓存标记落错行 + 账本身份碰撞的潜在误引用,不是整体不可用;若后续有生产上缓存命中塌方证据可升 p1)。

审核进行中:已在读 diff(src/incremental-lane.ts +38/-1,两个测试文件)、原 issue #5741、以及 applyTurn 全链路(resync/retouch/pin/marks)。完整八问结论随后跟帖。

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重: 不重复 —— 已对 #4599(CI 总控)查重:无既有条目处理本 PR 的红;继承红的根因已有哨兵修复(ci-fixes@0470bec62),按政策不为 CI 问题开新 issue。

标签: bug + scope/ci + p3(CI 继承红,仅内部流水线,零用户影响)

八问

  1. 是什么问题:PR [PR] fix(incremental): continue repeated-message identities across delta turns #5743(增量车道重复消息身份续接)的 11 个 test 分片全红,挡住合并。
  2. 什么原因导致的:不是本 PR 引入。master 0aace2f 自身 born-red——[PR] fix: unknown dsh one-shot sidecars steal the raw-key anchor and kill compression (context-explosion root cause) #5687(DSH: If the first request of a session originates from a bypass call, compression for the entire session is permanently | DSH: If the first request of a session originates from a bypass call, compress for the entire session is permanently inv | DSH:会话首个… #5656 单发 sidecar)合入后,五枚 dsh 回归钉仍假设 turn 1 走全量管线(sidecar residual 使 turn 1 改走逐字旁路),任何基于该 master 的 PR head 都继承这份红。我把 11 个分片的日志逐一扫过:21 处红全部映射到该签名(dsh guard-stamp 钉 ×12、[Bug] dsh 车道切模型 fork 出的 _sub: 会话不继承父会话 fold 块:上游上下文从 ~83K 瞬间膨胀到 794K、命中归 0、节省转负(0.1.187/0.1.188) #2817 stats off-by-one ×3、[Bug][dsh] Plugin lane performs a full re-parse and resends the entire body every round (81 MiB → hundreds of MB AST), w | [Bug][dsh] plugin 车道每轮全量解析重发 body(81 MiB→数百 MB AST)是 #5403 OOM 的生产者:用水位线断言替换逐条复核,服务端只解析新增尾段 #5418 A/B golden ×3),外加两枚已知时序 flake(msgid-lane-flow.test.ts:139 3 !== 4 settle race ×2 个 ubuntu 分片——与今日 [PR] fix(#5687): a tool-less MAIN turn must not be mistaken for a dsh sidecar - restores the BLOCKED-ref and wire-byte contracts #5709 同一签名、当时重跑即绿;wire-exit-matrix abort-openai-sse closed=0 ×1 windows 分片)。本 PR 只动 incremental/opencode 路径,不碰这些断言点;直接验证本 PR 行为的 msgid-lane 兄弟子测试全部通过。
  3. 影响范围与严重程度:只影响本 PR 及基于同一 master 的其他 PR head 的合并门禁(同继承红,各自 merge ci-fixes 即解);对用户零影响——修复是纯测试对齐,不碰生产代码。严重度:内部流水线阻塞。
  4. 怎么修复的:按 CI 协调政策吸收哨兵——把 origin/ci-fixes(0470bec,六个测试文件对齐 DSH: If the first request of a session originates from a bypass call, compression for the entire session is permanently | DSH: If the first request of a session originates from a bypass call, compress for the entire session is permanently inv | DSH:会话首个… #5656 已文档化行为)以独立 merge commit 并入 PR 分支并推送到 PR head 分支,新 head 763abe7,全矩阵重跑中。未自行改动任何生产代码或测试语义。
  5. 是否有回归问题:无。合并仅带入 6 个测试文件的对齐(哨兵已在同一 base 0aace2f 上本地验证六文件 + 相邻套件全绿);PR 自身 diff 原样未动;无语义漂移/阈值漂移/输出格式变化。残留风险:上述两枚 flake 可能在重跑中复现(msgid-lane-flow 本次在两个分片都挂),若重跑后仍红,留 attempt 2/3 继续处理。
  6. 是否破坏缓存:不涉及。
  7. 是否改了设计方向:无。
  8. 是否引入了新的配置:无。

判定:修复已推送(9c3faf3 → 763abe7),等 CI 重跑转绿;#4599 总控已留一行报备(inherited + ci-fixes merged + sha)。两枚 flake 若复现,attempt 2/3 继续。

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

审核进展(中间通报,完整八问结论随后):

已完成的核对

先不下结论,等定界证据齐了再发最终审核意见。

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 本轮完成(attempt 1/3):11 个 test 分片全红 = master 继承红(#5656 sidecar residual 的 dsh 回归钉,哨兵修复已在 ci-fixes@0470bec62)+ 两枚已知时序 flake(msgid-lane-flow :139 settle race ×2、wire-exit-matrix abort ×1),与本 PR diff 无关(逐分片扫过日志,21 处红全部映射到上述签名)。处置:merge origin/ci-fixes 进 PR 分支并推送,新 head 763abe7,全矩阵重跑中;#4599 已报备一行。等 CI 转绿;若两枚 flake 复现,attempt 2/3 继续。详细八问见上一条回复。

@ranxianglei

Copy link
Copy Markdown
Owner

(ai: vllm-qwen/qwen3.8-27b)

Heads-up: one test-only commit was added to this branch after review (e2fad486c, on top of the ci-fixes sentinel merge 763abe764):

  • test(rolling-marks): compare instances on substance, not wall-clock tag times
  • The cross-instance byte-equality assertions in tests/opencode-rolling-cache-marks.test.ts were flaky by construction: rendered ACP tags embed a wall-clock time attribute (kernel/src/render-refs.ts:72), so raw bytes across two independent proxy instances can never match once the runs straddle a second boundary. One local run showed 9 assertions (8 pre-existing from the compat commit + 1 new) failing on the seconds digit alone; every other byte matched.
  • Fix: strip the tag time="..." attribute before comparing; all other bytes still must match exactly. File now passes 12/12 twice locally; typecheck clean. CI on e2fad486c: 37 success / 14 skipped (gated e2e) / 0 fail.

No product behavior changed. Full review write-up with the eight-question conclusion is in the issue thread (#5741).

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

审核结论(结构化八问)— PR #5743,head e2fad486c

审核对象是当前 head,比接单时多了一个 commit:9c3faf3cb(产品修复)+ 763abe764(ci-fixes 哨兵合并,符合 CI 协调政策)+ e2fad486c(测试修复,见第 5 问)。查重结论已在首条回复给出:不重复。

  1. 是什么问题:增量车道(opencode V2 等插件宿主)每轮把 delta 独立转成 canonical 消息,「同内容重复计数器」每轮从零重启。宿主再次发送逐字节相同的内容(如用户连发 continue)时,新消息拿到与已确认前缀里旧孪生消息相同的 content ID——两条不同消息共用一个身份;更糟的是挂在新消息上的 cache mark 会写到旧孪生行上(cache_control 指向错误的前缀位置)。
  2. 什么原因导致的:anthropicToCore 按内容确定性派生 ID(hash 聚类);delta 独立转换丢失了「该内容在已确认前缀中已出现几次」的跨轮状态 → 计数器重启 → delta 的第一次出现撞上前缀已有序号。根因层在增量车道的跨轮簇状态不连续,不在 kernel(kernel 的 ID 铸造对完整历史是正确的,本 PR 未动 kernel)。
  3. 影响范围与严重程度:只影响走增量车道的插件宿主 + 跨轮重发逐字节相同内容(continue/ok 类重复短提示是典型场景;opencode V2 rolling cache mark 用户暴露面最大)。全程会话可用;后果为 (a) cache mark 落错位置 → 前缀缓存命中率受损,(b) 账本身份碰撞 → 按消息 ID 键控的下游 reconcile/decompress 存在潜在误归属。p2:无数据丢失、无不可用,触发条件窄但真实。
  4. 怎么修复的:每轮 delta 在参与身份校验前先过 continueContentClusters:对每个同内容簇,从「已确认前缀保留的最高序号 +1」续号(取 max 而非 count,稀疏恢复前缀不会回撞),保证 delta 的第 N 次出现拿到全局唯一的 (N+k) 号;重命名为单射(整簇加同一常数偏移,不产生新碰撞);cache mark 经同一 rename map 随迁。resync 仍走全新转换(正确——前缀重建即状态重置);客户端钉死 ID 保持优先(先改名、后校验/钉定,pin 撞到前缀任一 ID 照旧 400 拒绝)。正则与 kernel 实际铸造格式逐项核对一致(h_+sha256 前 16 hex,重复后缀 _N 从 1 起,kernel/src/wire/message-id.ts:50-58)。
  5. 是否有回归问题:触碰路径承担的既有行为逐项清点——代理模式/全量重发(恒转完整历史、计数器天然连续,未触碰)、resync 路径(行为不变)、retouch 路径(mark 直接写存储 span ID,不受改名影响,wireSpans 用改名后的最终 ID 建,代码验证)、pinning 路径(顺序保持 改名→校验→钉定,claimed 集含全部前缀 ID,代码+测试双覆盖)、冷恢复(稀疏前缀由 max-ordinal 规则覆盖,有专项单测)。高危模式逐项过:无 fail-fast 变吞错(新增 throw 仅当单簇重复超 2^53 才触发,实践不可达,且触发时与其他 applyTurn 错误同样降级 503 而非崩溃)、无阈值/时序漂移、出站格式变化仅限「重复内容消息拿不同 ID + mark 随行」(即修复本身)。发现一个真实缺陷、作者已在 PR 内修复:原 loopback 测试对两个独立代理实例做裸字节比较,而渲染出的 ACP tag 内嵌墙钟 time 属性——跨秒必挂(我在 9c3faf3cb 本地复现:12 个挂 9,diff 只差秒位数字);e2fad486c 比较前剥离 time 属性、其余字节仍严格相等,纯测试变更,我重跑 12/12 绿。残留(作者已披露、我认同):旧行为下已以碰撞 ID 写入的历史账本行不做修复,超出本 PR 范围。
  6. 是否破坏缓存:修复让 cache mark 落到意图行而非孪生行,是修复缓存指向,不是破坏;已确认前缀的字节从不被改写。另有一条性能观察(serialized-prefix 缓存的保守失效),单列决策评论,不阻塞本结论。
  7. 是否改了设计方向:否。架构不变(增量车道把转换后的 delta 追加到已确认前缀),只是在该架构内补上跨轮簇状态连续性;kernel ID 生成、身份层级(msgid > 顺序 > hash)、两种压缩模式均未动。
  8. 是否引入了新的配置:无。无 env、无 config 字段、无开关。

验证证据(本地定向,非全量;全量以远程 CI 为准)

  • typecheck(tsc --noEmit)@ e2fad486c 绿
  • tests/opencode-rolling-cache-marks.test.ts:12/12 通过(含新增六回合回归)
  • tests/incremental-content-clusters.test.ts + tests/incremental-lane.test.ts:27/27 通过(含 40 消息全切分差分对比)
  • 负控制(mutation):临时禁用 src/incremental-lane.ts:654 的跨轮续号 → 新回归测试失败;恢复 → 通过。回归测试有牙,不是假测试。
  • CI:叠层基线 [PR] fix(opencode): acknowledge V2 turns and preserve rolling cache marks #5742(b7f6e0b)全套绿(ubuntu/windows × node22/24 各 shard + static-gates);[PR] fix(incremental): continue repeated-message identities across delta turns #5743 最新 head e2fad486c 的 CI 结果尚未进镜像缓存(其前一 head 763abe764 各 gate 均绿),待远程裁定。
  • diff 整洁度:src/incremental-lane.ts +38/-1、新增单测文件、loopback +2 测试 + 比较器归一化;ci-fixes 哨兵合并合规;无 version/config/lockfile 变动;每一行都服务于本 PR 目的。

判定

方向正确(修根因非症状)、机制成立、测试有牙、diff 干净,代码层面可合。程序性注意两点:① 本 PR 叠在 #5742 之上(含其 commit),须先合 #5742,当前 mergeable_state=blocked 即由此而来;② 触碰 message-ref/id 逻辑,按 §7.4 属人工评审层,建议 owner 在 CI 对 e2fad486c 转绿后合并。另有一条性能观察需要决策,见下条评论。

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

🔶 需要决策
(ai: vllm-qwen/qwen3.8-27b)

背景一句话:审 #5743 时发现一个与叠层对(#5742/#5743)相关的性能取舍——修好身份碰撞后,「每轮有 mark 就保守失效 serialized-prefix 缓存」的旧逻辑对 rolling-mark 宿主变成了每轮全量重渲。

待决问题:这个精修要不要做、在哪做?

机制(给决策用的一句话):applyTurn 里 let marksTouched = flat.cacheControls.size > 0,只要本轮 delta 带任何 cache mark 就 bump marksVersion;而 marksVersion 一变,#5155 的「前缀已序列化缓存 + splice」优化整段失效,当轮退化为全历史 coreToAnthropic 重渲+序列化(O(n) 而非 O(Δ))。#5743 落地后,普通 delta 轮的 mark 结构性地只能落在新行(不能指向前缀行——这正是修复目标),所以这个 bump 对 rolling-mark 宿主(opencode V2,本特性的主要用户群)是纯过度失效:字节不变、白付 CPU。注意:粗失效本身由 #5742 引入,不是 #5743 新引入;#5743 只是让「mark 命中已有行」的情形消失,把成本完整暴露出来。正确性零影响。

选项:

我的推荐:B——bugfix PR 保持最小面先行合并;性能精修等数据说话再单开 PR(若数据显示占比可忽略则就此关闭,不浪费调度)。

不决定的默认行为:不动现有代码,#5743 正常走合并流程,此观察作为已知取舍记录在本线程。

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

CI 车道确认:新 head e2fad486c 全矩阵已绿(镜像缓存 14:32Z:13 success / 17 skipped(gated e2e)/ 0 fail),本地也核过该 commit 相对 763abe764 的 diff——仅 tests/opencode-rolling-cache-marks.test.ts(+12/-5),纯测试对齐,无产品代码变更,与通告一致。CI 侧无事可做,PR 就绪等 owner 合并。

@ranxianglei

Copy link
Copy Markdown
Owner

看起来有问题,这个增量车道不是按照msgid做的吗?怎么变成自己派发id了?

@ranxianglei ranxianglei left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(ai: vllm-qwen/qwen3.8-27b)

APPROVE — root-cause fix, arithmetic verified against the kernel's id-minting contract, regression test with teeth, clean diff. Extended analysis: linked issue thread (#5741). Structured conclusion below.

What it does. The incremental lane converts each delta turn independently, so anthropicToCore's per-pass content-cluster counter restarts at zero every turn: when the host re-sends byte-identical content (repeated continue prompts), the new row re-derives an ID an acknowledged row already owns, and the new row's cache mark lands on the older twin. continueContentClusters() continues each cluster's ordinal space from the acknowledged prefix (max retained ordinal + 1, sparse/folded-prefix safe), remaps cache marks through the same rename map; resync still starts fresh; client-pinned ids keep precedence (renaming runs before pin validation, claimed-id hard check unchanged). Kernel untouched.

Verified.

  • Offset arithmetic matches kernel minting exactly (h_ + sha256 first-16-hex, duplicate suffix _N; kernel/src/wire/message-id.ts:50-58); the rename is injective (constant per-cluster offset ⇒ cannot create a new collision).
  • Differential test over all 39 splits of a 40-message corpus: continued prefix+delta ids/marks deepEqual one full conversion; mutation negative control (disabling the continuation) fails the new regression test — the test has teeth.
  • Regression audit of touched paths: resync / retouch / msgIds pinning / boundary hash / cold rebuild all preserved; the only behavior change IS the fix (repeated content gets distinct continuing ids, mark on the intended row). Historical sessions' legacy colliding rows are NOT repaired (disclosed in the PR body; out of scope).
  • Head e2fad486c adds one test-only commit fixing a pre-existing structural flake: cross-instance raw byte comparison vs the wall-clock time="…" attribute embedded in rendered ACP tags (kernel/src/render-refs.ts:72) — strips that attribute only (precedent: the sanctioned normalization clause in #5403/#5418); every other byte still must match.

Procedural notes for the owner merge.

  1. Stacked PR — the head contains #5742's commits: merging this subsumes #5742 (close #5742 afterwards), or merge #5742 first.
  2. Branch is currently BEHIND master (6ddade163, #5700 T1 harvest-stall): no file overlap with this PR; auto-merge should be clean.
  3. Touches message-ref/id logic → §7.4 human-review tier: no auto-merge. CI on e2fad486c is green (13 success / 17 gated-skip / 0 fail) — ready for the owner to merge.
  4. PR is marked draft — un-draft when ready.
  5. One non-blocking perf observation is pending as 🔶 需要决策 in-thread (after the fix, rolling-mark hosts pay a full O(n) prefix re-render every marked turn because the coarse marksVersion invalidation was designed for the old collision case; options A/B/C with recommendation B = measure-first; default if undecided: ship as-is, zero correctness impact).

中文结构化八问

  1. 是什么问题:增量车道每轮独立转换 delta,同内容簇计数器每轮从零重启——宿主重发逐字节相同内容时,新消息复用旧孪生行的 content ID,且新消息的 cache mark 落到旧孪生行上。
  2. 什么原因导致的:跨轮簇状态不连续(delta 转换丢失「该内容在已确认前缀中已出现几次」);根因在增量车道而非 kernel(kernel 对完整历史的铸造是正确的,本 PR 未动 kernel)。
  3. 影响范围与严重程度:走增量车道的插件宿主(opencode V2 rolling-mark 暴露面最大)+ 跨轮重发相同内容;p2——会话全程可用,后果为 mark 错位→命中率受损、身份碰撞→下游潜在误归属。
  4. 怎么修复的:delta 参与身份校验前按「前缀保留最高序号+1」续号(稀疏恢复安全),mark 随同一 rename map 迁移;resync 全新转换与客户端钉死 ID 的优先级均不变。
  5. 是否有回归问题:resync/retouch/pinning/boundary/冷恢复逐项清点保持;唯一行为变更即修复本身(重复内容拿不同 ID + mark 落意图行,已披露);历史碰撞账本行不修(已披露)。测试侧发现一个既有结构性 flake(墙钟 time 属性致跨实例裸字节比较必挂),已由本分支 e2fad48 修掉,纯测试变更。
  6. 是否破坏缓存:否——修复的是 mark 指向错误的问题;已确认前缀字节从不改写。(性能观察另列决策,非正确性问题。)
  7. 是否改了设计方向:否——架构/身份层级(msgid>顺序>hash)/两种压缩模式/kernel 均未动。
  8. 是否引入了新的配置:无。

判定:代码层面可合;按 §7.4 属人工评审层,由 owner 合并(注意与叠层 #5742 的合入顺序)。

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Review lane 完成:GitHub review APPROVE 已提交(head e2fad486c)→ #5743 (review) 。CI 全绿(13 success / 17 gated-skip / 0 fail)。完整八问结论见 floor 14;本 lane 独立复核了全部关键论断(kernel 铸造格式对照、续号单射性、retouch/pinning/resync/boundary/冷恢复回归路径、差分测试与 mutation 负控制),无新增发现。flake 修复(e2fad48)独立核验成立:仅剥离墙钟 time 属性、其余字节仍严格相等,有 #5403/#5418 批准归一化条款先例,纯测试变更。

标签: bug/opencode/scope/wiring/p2 + pr + needs-decision(重判重打:severity 定为 p2——会话全程可用,后果为 mark 错位+身份碰撞潜在影响,与 floor 14 Q3 一致;scope 定为 wiring——缺陷在增量车道协议接线,scope/ci 系 CI 分片协调期的临时标,该工作已在 thread 完结;p3 移除;needs-decision 保留——性能观察 A/B/C 待 owner 裁决,推荐 B)。

无需顺手修项:flake 已由 ci-engineer-r1 在分支内修掉(e2fad48),其余审计未发现可当场修的小问题。剩余事项均为 owner 动作:① 合并(叠层 #5742 的合入顺序 + §7.4 人工评审层 + PR 当前 draft 需 un-draft);② 🔶 性能观察决策(floor 15;不决定的默认行为 = 现状发布,零正确性影响)。

ranxianglei pushed a commit to mariopasquali/billion-context that referenced this pull request Nov 5, 2026
…anxianglei#4542/etc (conflict resolution for ranxianglei#5743)

# Conflicts:
#	tests/opencode-rolling-cache-marks.test.ts
@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

冲突解决(本轮)

八问:

  1. 是什么问题:master 前移([PR] fix(opencode): acknowledge V2 turns and preserve rolling cache marks #5742/[PR] Index content's FK child columns — releaseContent DELETE stops full-scanning children #5540/[PR] fix(#4521): codex local auto-compaction rebase on the OpenAI wire lane #4542 等已合入)后,PR 分支与 base 冲突(merge state dirty),挡住合并。
  2. 什么原因导致的:[PR] fix(opencode): acknowledge V2 turns and preserve rolling cache marks #5742 是同一作者的兄弟 PR——与本 PR 共享 commit a6ea2a0af,且带同一枚 flake 的测试修复(fd625c8d2,canonMessages 归一化)。它先合进 master 后,本 PR 分支上独立实现的同类修复(our withoutTagTime)在 tests/opencode-rolling-cache-marks.test.ts 上与之撞出 4 处冲突。
  3. 影响范围与严重程度:仅本 PR 合并门禁;用户零影响(纯测试文件冲突 + 吸收 master 已合入变更)。严重度:内部流水线阻塞。
  4. 怎么修复的:merge origin/master(bfabcd1)进 PR 分支;4 处冲突全部取 master 侧 canonMessages——它注释明确遵循 [PR] relocate compress pair to its fold point | feat(compress): 压缩后把 compress 调用对挪到它自己的折叠点(+可选清未签名思考),两个默认关闭的实验开关 (#5470) #5471 的 sanctioned 归一化、已在 master 上,且比 ours 更精确(只掩时间戳形值、兼容转义形,不碰任意用户字符串);同时把本 PR 新增的两个测试也切到 canonMessages,删掉因此变成死代码的 withoutTagTime。相对 master 的净 diff = 恰好本 PR 自己的三个文件(src/incremental-lane.ts +38、两个测试 +34/+39),无夹带。已推送,新 head 21a676891。
  5. 是否有回归问题:无。merge 进来的全是 master 已验证变更(含 kernel kernel: T2/T3 distillation structurally starves when the T1 channel partially complies (#5671 treadmill root cause) | kernel: T2/T3 蒸馏在 T1 通道部分顺从时结构性饿死(#5671 跑步机根因) #5698 等,属 master 自身历史);PR 自身 diff 原样未动;冲突均为测试断言行,取 master 侧 = 用已过审的规范实现。无语义/阈值/输出格式漂移。
  6. 是否破坏缓存:不涉及。
  7. 是否改了设计方向:无。
  8. 是否引入了新的配置:无。

关于你的问题(msgid vs 自己派发 id)

车道仍是 msgid 优先,这个 PR 没有改那个层级。看 applyTurn 的执行顺序(src/incremental-lane.ts:654-670):

  • 先跑 continueContentClusters,处理的是 anthropicToCore 转换产物上的 h_<hash>[_n] 簇 id——这些是服务端转换生成的 bili 内部账本身份(模型线上从来没有消息级 id,KDD feat: self-update (auto + manual) #11),属于「content-derived identity」回退路径(MESSAGE-IDENTITY.md:identity is content-derived, not an assigned id);
  • 之后若宿主带了合法 msgIds,applyMsgIdPin 会把这些 id 整体覆写为宿主 msgid(cache marks 同步 re-key;重复认领直接 400)。即:有 msgid 时 msgid 恒赢,h_ id 只留在宿主没给 id 的行上(wire→core 非 1:1 展开的降级情形同理);
  • 本 PR 修的 bug 在回退路径自身:每个 delta 回合是一次全新转换,重复内容簇的 ordinal 每次从 0 重新起——同一 prompt 文本再次出现时会被分到一个已被前面确认行占用的 id(身份碰撞),副作用还会偷走旧行的 cache mark。修法是 ordinal 从已确认前缀续接(max+1),resync 时跳过。PR 自带的测试「continued content IDs can be replaced by client pins」钉的正是「客户端 pin 优先于续接的内容 id」。

判定:冲突已解并推送(新 head 21a676891),等 CI 全矩阵重跑;若有红继续处理。

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

状态更新:冲突已解并推送(merge origin/master bfabcd1 → 新 head 21a676891,4 处测试断言冲突全取 master 侧 canonMessages,净 diff 仍为本 PR 三个文件)。CI 已在新 head 上重跑:目前 10 success / 11 skipped / 0 fail,9 个 test 分片还在跑,暂无红。另有对「msgid vs 自派 id」问题的完整回答见上一条回复(结论:车道仍是 msgid 优先,本 PR 只修 content-derived 回退路径的 ordinal 碰撞)。等剩余分片出结果。

@ranxianglei ranxianglei left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(ai: vllm-qwen/qwen3.8-27b)

Re-review of merge commit 21a676891 (absorbs master bfabcd1ca) — APPROVE, verdict unchanged.

Delta vs the previously approved head (e2fad486c):

  • The merge pulls in master's recent work (#5540, #4542, #5698 T1 harvest-stall incl. its kernel bump, #5638, #5690, docs, and #5742 itself, which landed on master separately via 9a2d51d65). None of it is this PR's contribution.
  • The true PR payload vs current master is byte-identical to what I approved: git diff bfabcd1ca..21a676891 = the same 3 files (+110/-1). src/incremental-lane.ts and tests/incremental-content-clusters.test.ts are untouched by the merge (verified byte-equal to the reviewed commits).
  • One conflict resolution, in tests/opencode-rolling-cache-marks.test.ts: master's canonMessages() wall-clock-time masking (the same sanctioned cross-instance normalization, citing #4313/#5471) replaced the branch-side withoutTagTime() at all five comparison sites; the old helper is fully removed with no remnants, and both of this PR's regression tests ("repeated prompts retain distinct IDs…", "continued content IDs can be replaced by client pins…") survive and use the shared normalizer. Semantically equivalent (mask-to-constant vs strip, applied symmetrically to both A/B arms).
  • CI on 21a676891: 18 success / 12 gated-skip / 0 fail (mirror cache, fetched 17:20Z).

Prior procedural notes, updated: (1) the stacking dependency is now moot — #5742 is on master, so merging this PR lands exactly its own three-file diff; (2) the branch is current with master; (3)–(5) unchanged: §7.4 human-tier gate still applies, the PR is still marked draft, and the 🔶 performance observation (marksVersion over-invalidation, options A/B/C, recommendation B) remains open for owner decision.

Full original justification and 八问 stand in review 5787378806.

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

复审完成(第二轮):新 head 21a676891 → APPROVE(维持)。 review: #5743 (review) · #4599 已报备一行(6665600560)。

本轮变更与判定(八问速览,完整体例见上轮回复):

  1. 是什么:作者把 master 合入 PR 分支(merge commit),lane 重新触发复审。
  2. 原因:master 期间前进了([PR] fix(opencode): acknowledge V2 turns and preserve rolling cache marks #5742 已单独合入 master + [PR] Index content's FK child columns — releaseContent DELETE stops full-scanning children #5540/[PR] fix(#4521): codex local auto-compaction rebase on the OpenAI wire lane #4542/kernel: T2/T3 distillation structurally starves when the T1 channel partially complies (#5671 treadmill root cause) | kernel: T2/T3 蒸馏在 T1 通道部分顺从时结构性饿死(#5671 跑步机根因) #5698 T1/[PR] fix: stand acp_delegate down for user-scope pi-subagents installs too (#5637) #5638/[PR] fix: collapse adjacent output_config effort directives after fold reconstruction (#5688) #5690 等),分支需对齐。
  3. 影响:无新增影响面——对当前 master 的真实 payload 与上轮已审 diff 逐字节一致(3 文件,+110/-1);src/incremental-lane.ts、tests/incremental-content-clusters.test.ts 未被 merge 触碰。
  4. 修法:无新修复要审;唯一冲突解决在 tests/opencode-rolling-cache-marks.test.ts:跨实例 wall-clock time 归一化从分支侧 withoutTagTime() 切到 master 侧 canonMessages()(feat: Support precise timestamps for each message, allowing the model to recognize which messages are outdated and which | feat: 给每个消息支持精确时间,以让模型认识到什么消息是过期的什么是当前的新的 #4313/[PR] relocate compress pair to its fold point | feat(compress): 压缩后把 compress 调用对挪到它自己的折叠点(+可选清未签名思考),两个默认关闭的实验开关 (#5470) #5471 同一 sanctioned 归一化,mask-to-constant vs strip,双臂对称,语义等价),旧 helper 全删无残留;本 PR 两个回归测试均在且改用共享归一化。
  5. 回归:无——产品代码零变化;CI 于 21a676891 全绿(18 success / 12 gated-skip / 0 fail,镜像缓存 17:20Z)。
  6. 缓存:不涉及(与上轮同判)。
  7. 设计方向:不变;🔶 性能观察(marksVersion 过度失效,选项 A/B/C,推荐 B)仍挂起等 owner。
  8. 配置:不涉及。

顺带确认:上轮「#5742 须先合」的堆叠依赖已自然消解——#5742 已在 master(9a2d51d65),本 PR 现在就是独立三文件 diff,合并即落地全部贡献。

标签:维持 bug / opencode / scope/wiring / p2 / pr / needs-decision(状态未变,无需改标)。

收工判定:代码与 CI 均就绪,等 owner un-draft + 合并(§7.4 human tier);性能观察走 🔶 决策流,不阻塞合并。

ranxianglei pushed a commit to mariopasquali/billion-context that referenced this pull request Nov 6, 2026
…oth seams

ranxianglei#5741 redo, per maintainer direction: stop patching the derived-id symptom
(content clusters continued across delta turns) and give the lane what it was
designed to receive — the host's own session-scoped message ids.

What was wrong with the premise: opencode 2.x was assumed to expose no host
ids ("weakest of the four host seams"). It does: ctx.session.hook("context")
fires immediately before EVERY provider-request assembly carrying the full
{ id, role, content }[] conversation plus sessionID (verified live on
@opencode/cli 2.0.4, anthropic AND openai-compatible providers). V1 keeps
its experimental.chat.messages.transform seam as before; both now feed one
per-session store. With real ids riding, byte-identical repeated prompts get
distinct vouched ids (msg_r1 vs msg_r2), so the server-side pin creates one
row per occurrence and the collision class of ranxianglei#5741 disappears at the root.

Changes:
- opencode-v2.ts: register the "context" hook in createOpencodeV2Setup
  (shared by V1+V2 hosts; try/catch — hosts without the seam keep the
  documented id-less degrade; never fail startup over it)
- opencode-msgids.ts: deriveEntry now proves BOTH event shapes onto the wire
  1:1 with semantic part tokens (text bytes; tool call/result anchored by
  provider tool-use id, not input bytes; reasoning↔thinking; media↔image);
  host role "tool" normalizes to wire "user"; bookkeeping parts skip; ANY
  unrecognized shape still drops the whole field (宁可降级不可错位)
- tests/opencode-msgids.test.ts: tool-session end-to-end through the V2 seam,
  the ranxianglei#5741 repeated-prompt scenario (distinct fresh ids per occurrence),
  reasoning/thinking alignment, unknown-part and duplicate-id degrade paths

Server side needed NO changes: the claimed-set validation and pin remap
landed with ranxianglei#5742/ranxianglei#5743 already. continueContentClusters stays as the
fallback path for hosts/turns without a usable harvest.

Verified: tsc --noEmit clean; opencode-incremental/msgids/rolling-marks,
incremental-content-clusters, msgid-pin/algebra/lane-flow, dsh/pi-msgids,
incremental-lane, issue5403 fastpath, pins-wire-id-mints all green.
@ranxianglei

Copy link
Copy Markdown
Owner

增量车道改用真正的 host 消息 id(#5741 的修法按 owner 指示返工,2fa9dc47)

Owner 判定「修复机制不对:应该拿到真正的 sessionid+msgid,而不是派发消息」——已照此实现。此前分支里"跨 delta 延续派生 id"的那套是治标的:真正的问题是 opencode V2 根本没有把宿主自己的消息 id 送进车道,所以车道只能用内容哈希 h_… 兜底,字节相同的重复 prompt 才会撞车。现在堵的是根因。

前提纠正:以前认定 opencode 2.x 没有可见宿主 id 的接缝("四个 host 接缝里最弱的一个")。实测 @opencode/cli 2.0.4:ctx.session.hook("context") 在每次发 provider 请求前立刻触发,事件带完整会话 { id, role, content }[](role 有 user/assistant/tool)+ sessionID;anthropic 和 openai-compatible 两种 provider 都验证过。V1 原有的 experimental.chat.messages.transform 接缝保持不变。两条缝现在灌同一个 per-session store(每个 id 只存最后一次 fire,流式落定后同 id 内容增长天然覆盖)。

改动(392+/59-,4 文件):

  • src/agent/opencode-v2.ts:setup 里注册 "context" hook(try/catch 包裹,旧版本/无此接缝的宿主保持现状降级,绝不因此启动失败)
  • src/agent/opencode-msgids.ts:对证器升级为语义化 part token——文本比字节、工具调用/结果用 provider 的 tool-use id 作锚点(input 字节降低保真度不一,不比)、reasoning↔thinking、媒体↔image;宿主 role tool 归一到 wire 侧 user;step-start/snapshot 等纯宿主簿记 part 跳过;任何认不出的形态仍整体降级走派生 id(宁可降级不可错位,不变)
  • 测试:V2 接缝带工具会话端到端(seed 全量 id → ack 后 delta 只骑新 id)、Repeated content in incremental deltas reuses canonical message IDs #5741 场景直测(同一 prompt 两次出现分别 vouch msg_r1/msg_r2,pin 出两行,撞车从根上消失)、reasoning/thinking 对齐、未知 part 与重复 id 的降级路径
  • 服务端零改动:claimed-set 校验和 pin 重映射在 [PR] fix(opencode): acknowledge V2 turns and preserve rolling cache marks #5742/[PR] fix(incremental): continue repeated-message identities across delta turns #5743 原实现里已经就位;原来的 content-cluster 延续保留为「这次没采到可用 id」时的兜底路径

验证:tsc --noEmit 干净;opencode-incremental / opencode-msgids / rolling-cache-marks / incremental-content-clusters / msgid-pin / msgid-algebra / msgid-lane-flow / dsh-msgids / pi-msgids / incremental-lane / issue5403 fastpath / pins-wire-id-mints 全绿。

待办:CI 跑完后,之前挂起的 A/B 性能观察问题需要重新看一遍——真实 id 上车后缓存标记的落行行为和纯派生路径不完全一样,合并前建议补一轮真实 opencode 会话的端到端观察。PR 仍是 draft,合并动作留给人类。

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

CI 车道核对返工 head 2fa9dc47c:拓扑干净(建在我解冲突的 merge 21a676891 之上,base master bfabcd1ca);该 commit 的 diff 与报告逐字吻合——恰好 4 文件 +392/-59(opencode-v2.ts 注册 context hook、opencode-msgids.ts 对证器升级、opencode-native.ts、测试),服务端零改动(src/incremental-lane.ts 未动)。CI 已在新 head 全矩阵重跑中(检查时点:11 test 分片 + static-gates×2 + mysql-contract 在跑,0 fail)。合并前的「真实 opencode 会话端到端观察」待办已知悉——PR 是 draft、合并按约定留给人类,CI 结果出来我这边有红就继续处理。

ranxianglei pushed a commit to mariopasquali/billion-context that referenced this pull request Nov 6, 2026
… the V2 setup registration list

CI fix: setup now registers a second "context" seam (host-id harvest for
incremental msgIds) next to http.request, right before ws.handshake and the
report-filter loop's own "context"/compaction/title/generate entries.
@ranxianglei

Copy link
Copy Markdown
Owner

CI 修复($4c8a2b5a):上一版新增的 "context" 采集 hook 让 tests/opencode-native.test.ts:270 的注册表精确断言多了一项,已把期望列表更新为实际顺序(http.request → context(采集) → ws.handshake → context/compaction/title/generate(报表过滤))。三个红 shard(ubuntu-22/24、windows-22)都是这同一个失败,重试确认非 flake。其余 e2e/构建 workflow 在 $2fa9dc47 上已全部绿。

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

CI 车道核对修复 commit 4c8a2b5ad:diff 与报告一致——仅 tests/opencode-native.test.ts(+3/-1),把 V2 setup 注册表断言从 6 项更新为 7 项(http.request → context(采集) → experimental.ws.handshake → context/compaction/title/generate),并加了一行注释说明两个 "context" 各是什么;断言顺序与 opencode-v2.ts 实际注册顺序吻合,非放宽、非恒真。镜像缓存当前还停在旧 head 2fa9dc47(11 个 test 分片在跑,static-gates/mysql-contract 已绿);GitHub 识别新 head 后全矩阵会在 4c8a2b5ad 上重跑,有红我继续处理。

ranxianglei pushed a commit to mariopasquali/billion-context that referenced this pull request Nov 6, 2026
…ake, ranxianglei#5743 ubuntu-24 shard)

msgid lane flow: pin carries client ids... failed at tests/msgid-lane-flow.test.ts:139
with 'an edit lands as a FRESH row' 3 !== 4 on head 4c8a2b5 (run 42218432005).
Same signature as the two reds on head 9c3faf3 and PR ranxianglei#5709 — all went green
on rerun; sibling subtests and the other ten shards passed here. The store's
afterNextFlush verdict is scoped to writes currently queued or in flight, while
the lane wrap-up floats behind the ack (ranxianglei#5155/ranxianglei#3798), so the resync's fresh row
can settle one flush cycle after the first verdict on loaded runners. Bounded
re-probe (four settle cycles, 25 ms spacing > 15 ms debounce) before the strict
assert: a row that never lands still fails, just later — no weakening.
@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

CI 修复完成(新 head 76367874b)。八问:

  1. 是什么问题:head 4c8a2b5ad 上 1 个分片红(ubuntu-24,2),3154 个测试里仅 1 个失败:msgid lane flow: pin carries client ids... 在 tests/msgid-lane-flow.test.ts:139 断言 3 !== 4(「an edit lands as a FRESH row」)。
  2. 什么原因导致的:已知时序 flake——与本 PR 前一个 head(9c3faf3cb,两个分片)和 PR [PR] fix(#5687): a tool-less MAIN turn must not be mistaken for a dsh sidecar - restores the BLOCKED-ref and wire-byte contracts #5709 的失败签名逐字相同,历次重跑均绿。机制:store 的 afterNextFlush 判定只覆盖「当前已排队/在写」的写入(src/persist.ts:3057-3068),而车道收尾浮在 ack 之后(The incremental lane re-renders the entire conversation history and synchronously flushes it to disk each turn; large co | 增量车道每轮把整段会话历史重新渲染并同步落盘,大对话每轮多花约100毫秒 #5155/Incremental resync loses shared history during a cold canonical rebuild #3798,src/incremental-lane.ts:1036-1055),高负载 runner 上 resync 的新行可能在第一个判定之后隔一个 flush 周期才落地——测试的「一次判定即落定」假设比系统自身的持久化契约更严。非确定性佐证:同 head 另 10 个分片全绿、兄弟子测试全绿、返工 commit 只动 opencode agent 文件不碰此路径。
  3. 影响范围与严重程度:仅该测试在 CI 的可靠性(周期性红 + 消耗舰队 CI 预算 + 唤醒重复排查);用户零影响(纯测试改动)。严重度:内部流水线噪音。
  4. 怎么修复的:在该断言前加有界重探——最多 4 轮 settle(间隔 25ms > 15ms debounce),随后原样保留严格断言 assert.equal(rows.length, 4):行若永远落不了地,测试照样失败、只是晚几轮——不弱化任何断言,只把「落定窗口」对齐到系统真实契约。单点、测试专用、+12/-2。
  5. 是否有回归问题:无。生产代码零改动;断言的最终状态与之前完全一致;同文件其余 settle 读点有同类暴露但从未失败(若失败按同一机械手法处理,不预先扩散改动面)。
  6. 是否破坏缓存:不涉及。
  7. 是否改了设计方向:无。
  8. 是否引入了新的配置:无。

判定:flake 修复已推送(新 head 76367874b),等 CI 全矩阵重跑转绿。

@mariopasquali

Copy link
Copy Markdown
Contributor Author

(ai: openai-sub2api/gpt-6.1-sol)

@ranxianglei Could you review the current head and confirm what remains before this can be marked ready? Conflicts are resolved; CI has 36 passed and 13 skipped checks.

The native/WS jobs were skipped in CI. Our local native compression test also fails on unchanged master. Do you need further validation or a decision on the earlier cache-mark performance concern?

@ranxianglei

Copy link
Copy Markdown
Owner

Landed via batch PR #6482 (merged) — closing as superseded.

@mariopasquali

Copy link
Copy Markdown
Contributor Author

(ai: claude-opus-5-5)

This and #5954 were closed as landed via #6482, but #6482 is still open and not merged. Could they stay open until #6482 merges?

@ranxianglei ranxianglei reopened this Nov 14, 2026
ranxianglei pushed a commit to Louie0806/billion-context that referenced this pull request Nov 14, 2026
…oth seams

(content clusters continued across delta turns) and give the lane what it was
designed to receive — the host's own session-scoped message ids.

What was wrong with the premise: opencode 2.x was assumed to expose no host
ids ("weakest of the four host seams"). It does: ctx.session.hook("context")
fires immediately before EVERY provider-request assembly carrying the full
{ id, role, content }[] conversation plus sessionID (verified live on
@opencode/cli 2.0.4, anthropic AND openai-compatible providers). V1 keeps
its experimental.chat.messages.transform seam as before; both now feed one
per-session store. With real ids riding, byte-identical repeated prompts get
distinct vouched ids (msg_r1 vs msg_r2), so the server-side pin creates one
row per occurrence and the collision class of ranxianglei#5741 disappears at the root.

Changes:
- opencode-v2.ts: register the "context" hook in createOpencodeV2Setup
  (shared by V1+V2 hosts; try/catch — hosts without the seam keep the
  documented id-less degrade; never fail startup over it)
- opencode-msgids.ts: deriveEntry now proves BOTH event shapes onto the wire
  1:1 with semantic part tokens (text bytes; tool call/result anchored by
  provider tool-use id, not input bytes; reasoning↔thinking; media↔image);
  host role "tool" normalizes to wire "user"; bookkeeping parts skip; ANY
  unrecognized shape still drops the whole field (宁可降级不可错位)
- tests/opencode-msgids.test.ts: tool-session end-to-end through the V2 seam,
  the ranxianglei#5741 repeated-prompt scenario (distinct fresh ids per occurrence),
  reasoning/thinking alignment, unknown-part and duplicate-id degrade paths

Server side needed NO changes: the claimed-set validation and pin remap
landed with ranxianglei#5742/ranxianglei#5743 already. continueContentClusters stays as the
fallback path for hosts/turns without a usable harvest.

Verified: tsc --noEmit clean; opencode-incremental/msgids/rolling-marks,
incremental-content-clusters, msgid-pin/algebra/lane-flow, dsh/pi-msgids,
incremental-lane, issue5403 fastpath, pins-wire-id-mints all green.

(cherry picked from commit 2fa9dc4)
@ranxianglei
ranxianglei force-pushed the 2026-11-05_incremental-content-clusters branch from 805e142 to 2153152 Compare November 14, 2026 01:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Repeated content in incremental deltas reuses canonical message IDs

2 participants