Skip to content

[PR] Roll back the incremental canonical when the upstream forward fails | fix(lane): forward 失败后回滚增量 canonical,客户端重试同一 delta 不再被迫全量 resync - #4881

Open
ranxianglei wants to merge 3 commits into
masterfrom
2026-10-27_incremental-forward-rollback
Open

ranxianglei wants to merge 3 commits into
masterfrom
2026-10-27_incremental-forward-rollback

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

Closes #4867

问题

增量回合并入服务端 canonical(canonical 前进)之后,转发上游失败(抛异常,或完成但非 2xx)时 canonical 不回滚:客户端拿不到 ack 头(#3344 只在 2xx 打),下一回合用同一 delta 重试 → sinceSeq !== serverSeq → 409 seq-mismatch → 被迫全量 resync。每次上游 5xx/断流都白付一次整段历史重发。

修法(src/incremental-lane.ts)

  • applyTurn 之前对回合前状态做一轮快照:msgs 保留旧数组引用(applyTurn 赋的是新数组)、marks 克隆(applyTurn 原地改 Map)、外加 system/boundary 标量;
  • 异常路径:forward 包 try/catch——仅当「ack 不可能已到达客户端」(2xx 头未 flush)才回滚再抛出;若 2xx 头已发给客户端,ack 已生效,回滚反而造成反向漂移(客户端已推进、服务端没有);
  • 非 2xx 完成路径:按最终状态码判定(writeHead wrapper 里捕获),非 2xx 即回滚;其后既有 wrap-up 的指针登记自然落到旧分支行上,温缓存与冷重建保持一致;
  • resync 回合同样适用:快照恢复是 issue 所说「截断」的一般化(整体恢复被替换的前缀)。被拒分支残留在 append-only 表中的行由既有 409→resync 路径兜底(issue 划定的范围)。

边界说明:2xx 头已发出后流中断的场景,ack 视为已送达、不回滚——该场景本就不产生 409 循环,不在本次范围。

行为变更披露(旧值 → 新值)

  • 旧行为:forward 失败后,客户端重试同一 sinceSeq → 409 seq-mismatch → 全量 resync;
  • 新行为:同一 delta 直接重试成功(canonical 回到回合前长度)。成功路径行为完全不变;
  • 新增一条 warn 日志(回滚事件,含 conversationId 与 from/to 长度)。无新配置项、无 wire 协议变化、无缓存格式变化。

验证

  • 新增回归测试 tests/issue4867-incremental-forward-failure-rollback.test.ts(4 例:seed/delta/resync 失败后重试 + direct-call forward 抛异常):打补丁前全部红(精确复现 409 / canonical 停在推进后长度),打补丁后全部绿;
  • 既有增量车道相关测试 58/58 绿(incremental-lane / ack-pull / client-recovery / issue3304 / issue3343 / issue3433 / omp-native-incremental / opencode-incremental);
  • typecheck 干净;全量套件判定以 CI 为准。

@ranxianglei ranxianglei changed the title fix(lane): forward 失败后回滚增量 canonical,客户端重试同一 delta 不再被迫全量 resync | Roll back the incremental canonical when the upstream forward fails [PR] Roll back the incremental canonical when the upstream forward fails | fix(lane): forward 失败后回滚增量 canonical,客户端重试同一 delta 不再被迫全量 resync Oct 26, 2026
@github-actions

github-actions Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-27_incremental-forward-rollback (7662433)

Latest build — install this exact version

npm install -g billion-context@0.0.9400-pr.4881.133afe0

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Recent builds (commit → version)

commit version build
7662433 billion-context@0.0.9400-pr.4881.133afe0 build
6787c75 billion-context@0.0.9176-pr.4881.f50444b build
cec85bb billion-context@0.0.7196-pr.4881.681f1cf build
f679c7d billion-context@0.0.7117-pr.4881.223b391 build
d553129 billion-context@0.0.7112-pr.4881.415b233 build
a102b74 billion-context@0.0.6839-pr.4881.ed62c08 build
1421a8a billion-context@0.0.6368-pr.4881.bbea934 build
d2ca83c billion-context@0.0.6348-pr.4881.197f0bb build
3b8a7cd billion-context@0.0.6329-pr.4881.298f42a build
66ff3e0 billion-context@0.0.6323-pr.4881.b9960eb build

Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr4881.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

(ai: vllm-qwen/qwen3.8-27b) Merge-guard sync: origin/master (23158db) merged into this branch, pushed as 66ff3e0 — clean merge, zero conflicts (master touched no file this PR changes); PR now mergeable=true, CI re-running on the new head (all guard jobs green so far, test lanes in progress).

@ranxianglei

Copy link
Copy Markdown
Owner Author

(ai: vllm-qwen/qwen3.8-27b) Behind-alert resolved: no new master commits since this morning's sync (master tip still 23158db, branch head 66ff3e0 already contains it — mergeable=true, 0 behind); the repeated alerts came from a stale local mirror ref of this branch, now fast-forwarded to 66ff3e0 to match GitHub.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b) Merge-guard task closed out: (1) synced origin/master into this branch and pushed 66ff3e0 — clean merge, zero conflicts; (2) the repeated "behind" alerts turned out to be a stale local mirror ref of this branch (frozen at pre-sync 226c25c), which I fast-forwarded to 66ff3e0 after verifying its attached worktree was clean and idle. Final state: head 66ff3e0 contains all of master tip 23158db, mergeable=true, 0 behind, CI green on all guard jobs with test lanes completing. Nothing pending from me — ready for human merge once CI is fully green.

ranxianglei pushed a commit that referenced this pull request Oct 26, 2026
…already flushed)

Review supplement for #4881: the rollback gate in handleIncrementalTurn only fires while the ack cannot have reached the client, but the new suite only asserted the roll-back side. Add the negative-direction case: a forward that throws AFTER the 2xx head (carrying x-bili-applied-seq) was flushed must NOT roll back — the client already advanced its pointer, and a rollback would be reverse drift whose next delta 409s.
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重: 不重复 —— 检索 open+closed 的 issue/PR(canonical rollback / forward failure / 409 resync 家族),无同机制条目;本 PR 即 #4867 的唯一修复。

标签: bug + dsh/omp/opencode/pi(全部插件增量车道)+ scope/wiring(请求管道)+ p2(可用但每次上游抖动白付一次全量重发)。

独立审核结论

我没有照抄 issue 侧分析,重新读了 diff 与相关代码路径,并本地定向验证。

快照完整性(修法核心):对照 applyTurn 的实际写操作逐一核对——entry.msgs 是赋新数组([...base, ...deltaMsgs],旧引用有效)、entry.marks 是原地 set(克隆必需且已做)、boundary/system/systemSet 是标量重赋值——五个可写字段全覆盖,无遗漏。system 更新发生在 applyTurn 之后、快照窗口之内,回滚时一并恢复,正确。

门禁条件 vs 客户端契约:dsh-native.ts onResponse 确认客户端只在「200 + x-bili-applied-seq」时推进指针;200 无头被当作非标记响应(不推进),其他状态码保持原状态。因此「2xx 头未 flush 才回滚」的条件恰好等于「ack 不可能已到达客户端」——包括隐式头边界(headersSent 但 headStatus undefined,未打头→客户端不推进→回滚正确)。异常路径若 2xx 头已发出则不回滚,避免反向漂移,方向正确。

wrap-up 一致性:非 2xx 完成路径回滚后,recordCanonicalPointers(sessionId, entry.msgs) 用的是回滚后的数组,指针落到旧分支行,温缓存与冷重建一致。forward 前的持久化顺序(recordResyncBaseSeq/persistLaneState)先于本 PR 存在;重启窗口内的短暂漂移分析为无害(marks 多余条目无主、system 下轮无条件覆盖、失败的 resync 重试本身是整体替换,自愈)。

回归清点(逐项):成功路径行为不变;异常路径中 2xx 头已发出的场景前后都不回滚(不变);ack 头契约(#3344)不变;turn lock/pendingPtrWork 生命周期不变(finally 在异常路径完整释放);probe 端点在失败后报告回合前值(更诚实,且 adoption 另有 boundary hash 门,不会被污染)。未发现回归。

本地验证(定向):新测试文件 4/4 绿;既有 8 个增量车道套件(incremental-lane / ack-pull / client-recovery / issue3304 / issue3343 / issue3433 / omp-native-incremental / opencode-incremental)58/58 绿;typecheck 干净。上一 head(66ff3e0,含 master 合入)CI 全绿:test 全部分片(ubuntu/windows × node22/24)、static-gates、kernel-guard/lane-guard/config-surface-guard/version-guard/docs-sync/lock-sync/build-artifact 均 success。

Review 发现一处小问题 → 已直接修掉

回滚门禁只测了正向(seed/delta/resync 失败→回滚→重试落地;抛异常→回滚),负向没有钉住:「2xx 头已 flush 后抛异常 → 不回滚」这条分支无任何测试,违反仓库 §3 Gate Test Discipline(#3617:每个 gate 必须双向断言)。已在 PR 分支补上该用例(forward 在 writeHead(200) 后抛错 → probe 断言 canonical 保持推进长度 → 已采纳 ack 的客户端从 seq 2 继续无需 resync),独立标注 commit 3b8a7cd,本地 5/5 + typecheck 干净,新 head CI 排队中。

结构化八问

  1. 是什么问题:增量回合并入服务端 canonical 后,若转发上游失败,canonical 不回滚而客户端没拿到 ack,重试同一 delta 撞 409 seq-mismatch,被迫整段历史重发一次。
  2. 什么原因导致的:车道设计是「先入账再转发」(canonical 必须先前进,forward 才能发组装后的历史),但失败路径缺补偿——advance 不可逆 + ack 头只在 2xx 打,形成「服务端已推进、客户端未推进」的楔子。
  3. 影响范围与严重程度:全部插件增量车道(dsh/omp/opencode/pi);触发条件 = 回合进行中上游 5xx/断流;严重度为成本/体验(每次抖动多付一次全量重发 + 一次 409 往返),无数据丢失、无可用性影响 → P2。
  4. 怎么修复的:applyTurn 前对回合前状态做完整快照(msgs 旧引用 + marks 克隆 + 标量);失败路径按「ack 是否可能已到达客户端」补偿——异常路径仅 2xx 头未 flush 时回滚,非 2xx 完成路径按最终状态码回滚;wrap-up 指针登记自然落到旧行,温缓存与冷重建保持一致。
  5. 是否有回归问题:无。已审既有行为:成功路径、2xx 头已发出的异常路径、ack 头契约、turn lock 生命周期、probe 语义、指针/marks 持久化——逐项结论见上。
  6. 是否破坏缓存:否。回滚字节与客户端本地历史一致,下一轮出站前缀字节稳定;无去重键/游标/缓存格式变化。
  7. 是否改了设计方向:否。「先入账后转发」保留,回滚是同层补偿而非流程重排。
  8. 是否引入了新的配置:无。新增一条 warn 日志(回滚事件,含 conversationId 与 from/to 长度)。

判定:方向正确、实现完整、回归面受控、双向门禁现已齐备;唯一发现(负向测试缺失)已在 PR 分支修掉并推送。等 3b8a7cd 的 CI 转绿即可合并。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

CI 确认: 新 head 3b8a7cd(补负向门禁测试的 commit)的 CI 已全部通过——完整翻页核对全部 check runs(129 条、42 个不同检查项,按检查名取最新一次):28 success + 14 skipped,零红零挂起。期间出现的 cancelled 条目是同一 push 触发重复 run 被并发策略取消的旧副本,每个检查名的最新一次均为绿。PR mergeable=true,merge_state 显示可能略有滞后,以检查实际状态为准。

可以合并了。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b) Final merge-guard state: after the review lane's follow-up push, branch head is now 3b8a7cd (adds the missing negative-direction rollback-gate test) — verified locally it still fully contains master tip 23158db (0 behind, 3 ahead), so no further sync is needed from me. My work on this PR is complete: master sync merged+pushed (66ff3e0), stale local mirror ref converged, outcome comments posted. Handing off to the review lane + human merge once CI on 3b8a7cd is green. No further action expected from me on this PR.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b) Merge-guard sync #2: master had genuinely moved since the morning sync (23158db → 0cf53c8, +62 commits across the dsh/WS/anchor/catalog lanes) and none of them touch this PR's two files, so origin/master merged into the branch cleanly with zero conflicts — pushed as d2ca83c (now contains all of master: 0 behind, 4 ahead); PR mergeable=true, CI re-running on the new head. Also re-converged the stale local mirror ref of this branch (it had lagged at 66ff3e0 after yesterday's external push) so the merge guard's local view matches GitHub again.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b) Behind-alert #4 was genuine — synced: master had moved 0cf53c8 → e4cc685 (+5: hermes ACP session binding #4856/#4855 + docs/goldens, zero overlap with this PR's files); merged cleanly into the branch and pushed as 1421a8a (zero conflicts). Verified via API: head 1421a8a on base e4cc685, mergeable=true, checks green so far (20 success / gated lanes skipped / 2 queued); local mirror ref re-converged in step.

ranxianglei pushed a commit that referenced this pull request Oct 30, 2026
…back snapshot

Merging master brought #5155's per-turn render cache into Canonical
(renderPrefix + marksVersion). applyTurn bumps marksVersion and nulls
renderPrefix on resync, and laneItemsForTurn refreshes the prefix over the
NEW history before the forward lands. The #4867 pre-turn snapshot predates
those fields, so a rolled-back turn left a prefix cache rendered from the
rejected branch: if its len/marksVersion gates happened to match a later
turn's cut, laneItemsForTurn would splice rejected-branch bytes into an
accepted body. Snapshot and restore both fields.
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Both new "conflict" triggers verified stale — no action taken: head f679c7de2 already fully contains live master (44067a5bc, test-merge = "Already up to date", zero conflicts); current state is mergeable with no conflict, blocked only by pending CI (3 pending / 0 failed). Superseding my previous outcome comment above.

ranxianglei pushed a commit that referenced this pull request Oct 30, 2026
…cold-tail storage-mode pin

Clears the inherited #5140 cold-restored-parent raw-tail CI red on this branch.
Net delta vs f679c7d is exactly tests/plugin-fork.test.ts (+9/-1): pin
BILI_STORAGE_UNIFIED="full" for the raw-tail proof (the legacy runner baseline has
no unified rows so forkSnapshot fails closed with 409 SNAPSHOT_UNAVAILABLE) and restore
the ambient value in finally. Docs/config registrations from ci-fixes were already
present via master@44067a5bc. No source changes.
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

CI 修复(auto-fix 车道):本轮 6 个红分片全部是继承自 master 的 #5140 冷尾 fork flake(叠加 Windows 环境 flake 家族),非本 PR 代码。#5140 现已在 ci-fixes 定位修复(一个测试存储模式钉),我已将该修复合入本分支并推送。

证据(逐条可核)

#5140 根因(已在 ci-fixes 定位修复):该用例证明 forkSnapshot 能从 UNIFIED 索引重建「非保留」父会话的冷恢复 raw tail,但用例自身没钉存储模式;CI 全局基线 BILI_STORAGE_UNIFIED ??= "legacy" 使 broad corpus 跑在 legacy → 无 unified 行 → 重建无源 → forkSnapshot 正确 fail-closed 返回 409。修法 = 单用例显式钉 BILI_STORAGE_UNIFIED="full" + finally 还原 ambient(纯测试 diff,+9/-1,非掩盖)。修复体在 ci-fixes@67f8e894。

本次动作:本分支(f679c7d,含 master@44067a5bc)缺这个测试钉,故仍撞 #5140。已按 CI 协调纪律 git merge origin/ci-fixes@67f8e894 进分支——干净自动合并、零冲突,净变更仅 tests/plugin-fork.test.ts(+9/-1),零源码改动(ci-fixes 的文档/config 登记经 master@44067a5bc 早已在位)。已推送新 head cec85bbbe,CI 重跑中,预期 6 片的 #5140 全部转绿。

结构化八问

  1. 是什么问题:PR [PR] Roll back the incremental canonical when the upstream forward fails | fix(lane): forward 失败后回滚增量 canonical,客户端重试同一 delta 不再被迫全量 resync #4881 的 CI 有 6 个 test 分片红。
  2. 什么原因导致的:全是继承自 master 的 [Bug][fork] Cold restore cannot use an intact unified raw tail for a non-retained parent (#4266 item 4) #5140 冷尾 fork 用例(legacy 存储基线下确定性 409)+ Windows 环境 flake 家族(promptCache.routing is ignored by relay-affinity stamping; no way to persist per-endpoint prompt_cache_key abatement (0.1.209) #5103 / fix: bound ordinary session loading and avoid full reloads for lazy fields #4739 EBUSY / resident-minimal hang,受害分片逐次漂移);本 PR diff(src/incremental-lane.ts + 回归测试)与失败路径零交集。
  3. 影响范围与严重程度:对 [PR] Roll back the incremental canonical when the upstream forward fails | fix(lane): forward 失败后回滚增量 canonical,客户端重试同一 delta 不再被迫全量 resync #4881 = 阻断合并门禁;根因是 master 级 [Bug][fork] Cold restore cannot use an intact unified raw tail for a non-retained parent (#4266 item 4) #5140 + Windows 环境项,波及当前所有基于近期 master 的 PR(CI Control Center: Centralized CI Failure Handling & Status Sync (Comment all CI issues here; do not open new issues) | CI 总控: CI 失败集中处理与状态同步(所有 CI 问题在此评论, 勿开新 issue) #4599 今天在案几十个同签名)。本 PR 自身代码无缺陷、无回归。
  4. 怎么修复的:不碰本 PR 代码;把 ci-fixes 里已定位好的 [Bug][fork] Cold restore cannot use an intact unified raw tail for a non-retained parent (#4266 item 4) #5140 测试钉合进分支(仅 plugin-fork.test.ts +9/-1),让该用例在它本就要求的存储模式下运行。
  5. 是否有回归问题:无——只在一个测试内加了存储模式钉 + finally 还原,不改任何 src/;merge-guard 的 The incremental lane re-renders the entire conversation history and synchronously flushes it to disk each turn; large co | 增量车道每轮把整段会话历史重新渲染并同步落盘,大对话每轮多花约100毫秒 #5155 扩展经 CI 验证通过;干净合并、零冲突、零源码改动。
  6. 是否破坏缓存:不涉及(纯测试夹具,不动运行时)。
  7. 是否改了设计方向:否。
  8. 是否引入新配置:无(未新增任何 config/env;只是让既有 [Bug][fork] Cold restore cannot use an intact unified raw tail for a non-retained parent (#4266 item 4) #5140 用例显式声明它本就需要的存储模式)。

收工判定:本 PR 内容就绪、可合;本轮已把继承的 #5140 修复合入并推送(cec85bbbe),CI 重跑中——预期 #5140 六片转绿。若 Windows 分片仍残留 #5103 / #4739 / resident-minimal 等环境性 flake,那属 master 级 base-code 项,需 owner 在 #4599 triage(已在档),不在本 PR/本车道代修。另:本分支与最新 master 的同步/冲突归 merge-guard 车道(它刚在 f679c7d 处理过)。

@ranxianglei
ranxianglei force-pushed the 2026-10-27_incremental-forward-rollback branch from cec85bb to 6787c75 Compare November 11, 2026 17:10
ranxianglei pushed a commit that referenced this pull request Nov 11, 2026
…already flushed)

Review supplement for #4881: the rollback gate in handleIncrementalTurn only fires while the ack cannot have reached the client, but the new suite only asserted the roll-back side. Add the negative-direction case: a forward that throws AFTER the 2xx head (carrying x-bili-applied-seq) was flushed must NOT roll back — the client already advanced its pointer, and a rollback would be reverse drift whose next delta 409s.

(cherry picked from commit 3b8a7cd)
ranxianglei pushed a commit that referenced this pull request Nov 11, 2026
…back snapshot

Merging master brought #5155's per-turn render cache into Canonical
(renderPrefix + marksVersion). applyTurn bumps marksVersion and nulls
renderPrefix on resync, and laneItemsForTurn refreshes the prefix over the
NEW history before the forward lands. The #4867 pre-turn snapshot predates
those fields, so a rolled-back turn left a prefix cache rendered from the
rejected branch: if its len/marksVersion gates happened to match a later
turn's cut, laneItemsForTurn would splice rejected-branch bytes into an
accepted body. Snapshot and restore both fields.

(cherry picked from commit d553129)
ci-check added 3 commits November 13, 2026 04:44
…ward fails

applyTurn advanced the warm canonical BEFORE the assembled request was forwarded upstream; when that forward failed (thrown transport error, or a completed non-2xx), the canonical stayed ahead while the client received no ack (x-bili-applied-seq is stamped on 2xx only, #3344) — so the client's plain retry of the SAME delta hit a 409 seq-mismatch and paid a full resync. Every upstream blip became one full-history resend.

Snapshot the pre-turn state (msgs ref, CLONED marks map, system/boundary scalars) right before applyTurn and restore it when the forward does not land:
- thrown forward: roll back only while the ack cannot have reached the client (a flushed 2xx head means the client already advanced on it);
- completed non-2xx / headless end: roll back — no ack went out.
Append turns truncate back to the pre-turn length; resync turns restore the replaced prefix whole (generalizing truncation). Rows of the rejected branch stay in the append-only table — the existing 409->resync path owns them. On the resolved-non-2xx path the wrap-up chain now records pointers onto the restored old-branch rows, keeping the warm and cold views consistent.

Regression tests (all red before the fix, green after): tests/issue4867-incremental-forward-failure-rollback.test.ts — seed/delta/resync retry-after-failure over HTTP plus the direct-call thrown-forward case.

(cherry picked from commit 226c25c)
…already flushed)

Review supplement for #4881: the rollback gate in handleIncrementalTurn only fires while the ack cannot have reached the client, but the new suite only asserted the roll-back side. Add the negative-direction case: a forward that throws AFTER the 2xx head (carrying x-bili-applied-seq) was flushed must NOT roll back — the client already advanced its pointer, and a rollback would be reverse drift whose next delta 409s.

(cherry picked from commit 3b8a7cd)
…back snapshot

Merging master brought #5155's per-turn render cache into Canonical
(renderPrefix + marksVersion). applyTurn bumps marksVersion and nulls
renderPrefix on resync, and laneItemsForTurn refreshes the prefix over the
NEW history before the forward lands. The #4867 pre-turn snapshot predates
those fields, so a rolled-back turn left a prefix cache rendered from the
rejected branch: if its len/marksVersion gates happened to match a later
turn's cut, laneItemsForTurn would splice rejected-branch bytes into an
accepted body. Snapshot and restore both fields.

(cherry picked from commit d553129)
@ranxianglei
ranxianglei force-pushed the 2026-10-27_incremental-forward-rollback branch from 6787c75 to 7662433 Compare November 12, 2026 21:01
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷

Conflict resolved — PR #4881 now merges cleanly against current master.

What I did this run:

  1. Diagnosed the dirty state. The PR branch (2026-10-27_incremental-forward-rollback, old head cec85bbbe) was based on a stale master and the history has criss-cross multi-merge-bases. A plain git merge master produced ~90 spurious conflicts (git's 3-way base selection is confused by the tangled bases), so I used a clean reconstruction: replay only the PR's real contribution onto the current master tip.

  2. Isolated the true contribution. The PR's real surface is exactly 3 commits: the Incremental lane: After upstream forwarding fails, the client incurs an unnecessary full re-sync. | 增量车道:转发上游失败后,客户端白付一次全量重同步 #4867 rollback fix (src/incremental-lane.ts + new tests/issue4867-incremental-forward-failure-rollback.test.ts), the negative-gate test, and the The incremental lane re-renders the entire conversation history and synchronously flushes it to disk each turn; large co | 增量车道每轮把整段会话历史重新渲染并同步落盘,大对话每轮多花约100毫秒 #5155 renderPrefix/marksVersion coverage. (The plugin-fork.test.ts [Bug][fork] Cold restore cannot use an intact unified raw tail for a non-retained parent (#4266 item 4) #5140 storage pin was already in master — no re-apply needed.)

  3. Caught a stale base mid-run. My first rebase landed on a stale master (33104b1f95) because the local origin/master ref is ambiguous in this repo and resolved to an old local branch. The authoritative GitHub master had actually advanced to 259cd87674 (128 commits ahead, incl. a kernel bump to 0.0.196 and a new ingestRowsNow durability call in the very same file). I re-based onto the true master.

  4. Resolved the conflict. In src/incremental-lane.ts, the pre-turn snapshot region had to merge my Incremental lane: After upstream forwarding fails, the client incurs an unnecessary full re-sync. | 增量车道:转发上游失败后,客户端白付一次全量重同步 #4867 preTurn snapshot (with renderPrefix/marksVersion) with master's new retouch argument to applyTurn; master's ingestRowsNow call in the settlement block coexists untouched. rollbackCanonical restores all 7 fields.

  5. Verified (targeted, local). typecheck clean against kernel 0.0.196; targeted suites green: issue4867-incremental-forward-failure-rollback 5/5, incremental-lane + issue5155-incremental-per-turn 32/32 (37 total). The full-suite verdict is left to remote CI on the pushed head (per local-test discipline I did not run the full suite locally).

  6. Force-pushed (the authorized conflict-rebase, --force-with-lease on my prior push). New head 76624335b4.

Live state (checked via authoritative git ls-remote, since the mirror cache was stale at cec85bbbe): master 259cd87674 is now an ancestor of the PR head → merge state clean, no conflicts. The PR diff vs master is exactly src/incremental-lane.ts (+44/−1) and the new 271-line regression test.


八问(中文)

  1. 是什么问题:PR [PR] Roll back the incremental canonical when the upstream forward fails | fix(lane): forward 失败后回滚增量 canonical,客户端重试同一 delta 不再被迫全量 resync #4881 与目标分支 master 冲突(merge state: dirty),无法合并。
  2. 什么原因导致的:PR 分支基线过旧,且历史存在 criss-cross 多 merge-base,git 三方合并基选择被搅乱,普通 merge 产生约 90 个虚假冲突。根因是基线过旧 + 历史纠缠,不是代码本身有真实冲突。
  3. 影响范围与严重程度:仅影响该 PR 的合并能力(无法 merge),不影响任何已发布行为;属合并阻塞,非运行时缺陷。
  4. 怎么修复的:不做整包 merge,而是「干净重建」——只把 PR 真实的 3 个提交(Incremental lane: After upstream forwarding fails, the client incurs an unnecessary full re-sync. | 增量车道:转发上游失败后,客户端白付一次全量重同步 #4867 回滚修复 + 负向门测试 + The incremental lane re-renders the entire conversation history and synchronously flushes it to disk each turn; large co | 增量车道每轮把整段会话历史重新渲染并同步落盘,大对话每轮多花约100毫秒 #5155 覆盖)rebase 到当前 master 尖端;冲突点(src/incremental-lane.ts 的 preTurn 快照区)手工合并,保留我方回滚逻辑,同时并入 master 的 retouch 参数与 ingestRowsNow 持久化调用。
  5. 是否有回归问题:无。typecheck 干净;定向测试 37/37 全绿(issue4867 回滚 5/5、incremental-lane + issue5155 32/32);既有行为(回滚门、The incremental lane re-renders the entire conversation history and synchronously flushes it to disk each turn; large co | 增量车道每轮把整段会话历史重新渲染并同步落盘,大对话每轮多花约100毫秒 #5155 前缀缓存、Compression fails in long sessions while new sessions work fine | 压缩失败.新会话反而没事 #3397 持久化)逐项核对共存正确。全量回归交由远程 CI 在推送头判定。
  6. 是否破坏缓存:不涉及——本次只解决合并冲突,未改动任何缓存/去重/游标逻辑;The incremental lane re-renders the entire conversation history and synchronously flushes it to disk each turn; large co | 增量车道每轮把整段会话历史重新渲染并同步落盘,大对话每轮多花约100毫秒 #5155 的 renderPrefix/marksVersion 回滚覆盖原样保留。
  7. 是否改了设计方向:不涉及——未改架构/边界/数据流,仅把既有 PR 提交落到新基线。
  8. 是否引入了新的配置:不涉及——无新增 env/config/默认值。

判定:冲突已解决,PR 现可干净合并(master 为 PR 头祖先,merge state 转 clean);全量 CI 绿/红以推送头 76624335b4 上的远程 CI 为准(本次未本地跑全量)。收工。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

查重: 不重复 — #4881 是 #4867 的唯一定向修复 PR; 本轮是 ci-engineer 干净重建后 head 76624335b4 的独立复审 (与 floor 24 首轮审核同一 PR, 非新工作项)。

本轮审查范围: 重建后的 head (3 个真实 commit 重放到 master 259cd87674 之上), diff = src/incremental-lane.ts (+44/−1) + 新测试文件 (271 行, 5 测试)。

独立核查结论 (新 head):

  1. rebase 状态: 分支落后 master (99c22d283f) 50 个 commit, 但其中没有一个触碰 src/incremental-lane.ts 或新测试文件; git merge-tree = 0 冲突; 新增 master commit (docs/announcements/BILI_PLUGIN_BYPASS/preflight/watchdog/responses/storage-shadow) 与 PR 无交互。
  2. CI 全绿 (76624335b4, 48 runs): 全部 test shards (ubuntu 22/24 ×3, windows 22 ×5) success; 全部 guard lanes (static-gates, kernel-guard, lane-guard, config-surface-guard, version-guard, docs-sync, lock-sync, build-artifact…) success; e2e lanes gated-skip。
  3. 冲突合并正确性: preTurn 快照正确地把 master 的 renderPrefix/marksVersion (The incremental lane re-renders the entire conversation history and synchronously flushes it to disk each turn; large co | 增量车道每轮把整段会话历史重新渲染并同步落盘,大对话每轮多花约100毫秒 #5155) 与 PR 原有的 5 个字段合并为 7 字段快照/恢复; master 的 applyTurn retouch 参数与 ingestRowsNow 持久化调用 (Compression fails in long sessions while new sessions work fine | 压缩失败.新会话反而没事 #3397) 在同文件中完整保留, 语义未被合并破坏。
  4. 回滚语义独立复核: 异常路径仅在 2xx 头未 flush 时回滚 (负向门保留, 测试 5 钉住); 完成路径按 headStatus 非 2xx/缺失回滚 (Interval review – residual minor fix list: two stale doc entries, kernel lock file lag, missing INERT alert on delete config, SIGKILL escalation, ack header 2xx-only (7 items, all minor) | 区间审查残留小修清单:文档过期两处、kernel 锁文件滞后、删除配置缺 INERT 告警、SIGKILL 升级、ack 头 2xx-only(7 项,均为小修) #3344 ack 契约: ack 只在 2xx 盖); 回滚后 wrap-up — ingestRowsNow 持久化的是回滚后的 entry (被拒分支的行留在 append-only 表, 由既有 409→resync 路径接管, 设计如此); recordCanonicalPointers 在 flush 后读 LIVE entry → 指针落在旧分支行; 锁在 finally 释放 (!settled 分支)。
  5. 回归清点 (既有行为逐项): 成功路径 (2xx 不回滚, 不变) / 2xx 已 flush 的异常 (不回滚, 负向测试钉住) / ack 契约 (Interval review – residual minor fix list: two stale doc entries, kernel lock file lag, missing INERT alert on delete config, SIGKILL escalation, ack header 2xx-only (7 items, all minor) | 区间审查残留小修清单:文档过期两处、kernel 锁文件滞后、删除配置缺 INERT 告警、SIGKILL 升级、ack 头 2xx-only(7 项,均为小修) #3344) / turn 锁 / probe 语义 / 指针与 marks 持久化 / The incremental lane re-renders the entire conversation history and synchronously flushes it to disk each turn; large co | 增量车道每轮把整段会话历史重新渲染并同步落盘,大对话每轮多花约100毫秒 #5155 前缀缓存 (renderPrefix 随快照恢复) / Compression fails in long sessions while new sessions work fine | 压缩失败.新会话反而没事 #3397 持久化 — 全部复核, 无意外回归。

本轮新发现 (1 处, 正在 PR 分支直接修): wireSpans 漏出快照/回滚。

  • Canonical 共 9 个字段, applyTurn 写其中 8 个 (sessionId 不可变); 快照覆盖 7 个 — wireSpans 是唯一被 applyTurn 写入却漏掉的字段 (src/incremental-lane.ts:656: entry.wireSpans = resync ? deltaSpans : [...entry.wireSpans, ...deltaSpans], 每轮赋新数组)。
  • 唯一消费者是 retouch 校验 (src/incremental-lane.ts:598): spans[r.index] → byId → sameWireContent → 409 "retouch index N content differs"。
  • 故障序列 (append): 失败的 delta D → 回滚后 spans=S0+SD (残留); 重试 D → S0+SD+SD; 再下一个 delta E → S0+SD+SD+SE; 客户端对 E 内消息的 retouch 读到 D 的残留 span → 内容比对失败 → 假 409 → 全量 resync (正是本 PR 要消灭的成本, 经 retouch 面复活); 且每轮失败残留一段 span, 只增不减。
  • 故障序列 (resync): 失败的 resync → spans 整体变成被拒替换的 spans; 客户端按旧长度发 delta → spans 完全错位; 对旧 canonical 的 retouch 409 直到下次 resync 自愈。
  • 严重度: 低-中 — 仅 retouch 面 (cache_control 装饰优化, dsh/omp/opencode/pi 滚动断点); 故障模式 = 假 409 → 文档化的 resync 恢复; 无数据丢失/损坏。PR 之前残留 spans 不可达 (根本没有回滚), 此缺口由 PR 的不完整快照引入。
  • 修复 (2 行 + 2 个回归测试, 正在推): preTurn 加 wireSpans: entry.wireSpans; rollbackCanonical 加 entry.wireSpans = preTurn.wireSpans (安全 — applyTurn 只赋新数组, 不改原数组); 测试钉住 append 错位与 resync 错位两种序列下 retouch 必须 200 而非假 409。

八问:

  1. 是什么问题: Incremental lane: After upstream forwarding fails, the client incurs an unnecessary full re-sync. | 增量车道:转发上游失败后,客户端白付一次全量重同步 #4867 — 上游 forward 失败时, 增量车道已在 forward 之前推进了服务端 canonical, 客户端原样重试同一 delta 会撞 409 seq-mismatch, 被迫全量 resync (整段历史重发 + 前缀缓存失效)。
  2. 什么原因导致的: 执行顺序 — applyTurn (canonical 前进) 先于 forward (上游投递), 而 forward 失败时没有恢复路径; ack 契约 (Interval review – residual minor fix list: two stale doc entries, kernel lock file lag, missing INERT alert on delete config, SIGKILL escalation, ack header 2xx-only (7 items, all minor) | 区间审查残留小修清单:文档过期两处、kernel 锁文件滞后、删除配置缺 INERT 告警、SIGKILL 升级、ack 头 2xx-only(7 项,均为小修) #3344) 在非 2xx 时已经不发 ack, 客户端不知道服务端已前进, 两边从此错位。
  3. 影响范围与严重程度: 所有开了增量车道的插件会话 (dsh/omp/opencode/pi) 在上游抖动时触发; 修复前每次抖动 = 一次强制全量 resync (成本尖峰), 能用但有明显代价; 本轮发现的 wireSpans 缺口: 低-中, 仅 retouch 面, 假 409 后 resync 自愈, 无数据影响。
  4. 怎么修复的: applyTurn 前对 pre-turn canonical 做快照; forward 失败 (throw 或完成时非 2xx) 且 2xx 头未 flush (ack 不可能已到达客户端) 时恢复快照; 被拒分支的行留在 append-only 表由既有 409→resync 路径接管。本轮补: wireSpans 进快照/恢复 + 2 个回归测试。
  5. 是否有回归问题: 无。逐项清点: 成功路径不变 (2xx 不回滚)、2xx 已 flush 不回滚 (负向门测试钉住)、ack 契约/turn 锁/probe 语义/指针与 marks 持久化/The incremental lane re-renders the entire conversation history and synchronously flushes it to disk each turn; large co | 增量车道每轮把整段会话历史重新渲染并同步落盘,大对话每轮多花约100毫秒 #5155 前缀缓存/Compression fails in long sessions while new sessions work fine | 压缩失败.新会话反而没事 #3397 持久化均未变; 唯一行为变更 = 失败 forward 现在会恢复 canonical (即本 PR 目的) + wireSpans 快照完整性 (本轮)。
  6. 是否破坏缓存: 不破坏, 方向相反 — 回滚时 renderPrefix 一并恢复, 下轮请求按 pre-turn 前缀重渲染 (字节稳定); 本 PR 消灭的是 resync 成本。wireSpans 缺口原本会在 抖动+重试+下一 delta 后于 retouch 面制造假 409→resync (缓存失效事件), 已随本轮修复消除。
  7. 是否改了设计方向: 没有。同一车道、同一 ack 契约、同一 409→resync 兜底, 只在失败分支加恢复路径。
  8. 是否引入了新的配置: 没有。

判定: 方向正确, 机制成立, 回归清点干净, CI 全绿 — 可合; 合并前等本轮 wireSpans 修复 commit (2 行 + 2 测试) 推上去 CI 再绿一次。推完我在本线程跟进。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

状态更新 (接上一条完整审核结论): wireSpans 修复的 2 行改动已在本地 PR 分支工作区完成 (preTurn 快照 + rollbackCanonical 各加一行), 但尚未推送 — 我新加的两个回归测试 (append 错位 / resync 错位下的 retouch 校验) 本地跑出来是 409 而非预期的 200, 原有 5 个测试全部通过。

409 的响应体不带 detail (handle 路径在 src/incremental-lane.ts:892 把 applied.detail 丢掉了, 只进日志), 且失败轮的 sinceSeq === serverSeq, 所以 409 不是 sinceSeq 门 (line 587), 而是 retouch 校验内部 (line 599/603/605/610 之一) — 说明我对 wireSpans 错位机理的理解还差一环 (可能是 core id 的稳定性或 retouch 消息转换的细节), 正在用临时 debug 行定位精确原因。

不会推红测试: 定位清楚之前 commit 不推。定位后两种结局: ①我的测试构造有问题 → 修测试, 2 行修复照推; ②修复本身还有缺口 → 扩大修复范围再推。两种都会在本线程跟进完整结论。

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

1 participant