Skip to content

fix: attribute incremental-turn acks by per-request turn id, not URL (#3329) - #3340

Merged
ranxianglei merged 1 commit into
masterfrom
2026-10-16_fix-incr-ack-url-keying
Oct 17, 2026
Merged

ranxianglei merged 1 commit into
masterfrom
2026-10-16_fix-incr-ack-url-keying

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

Closes #3329

What is broken

The pending-ack maps of the incremental-turn feature (#3248) in opencode-native, omp-native and dsh-native were keyed by request URL. Two conversations sharing one provider URL could cross-attribute x-bili-applied-seq when a response arrived late (or its slot was evicted): the true owner's ack was lost, both watermarks got poisoned, and the sticky dead flag killed incremental turns for BOTH conversations permanently (both fall back to full resend forever).

Fix

Per-request correlation id; zero server-side changes:

  1. native-intercept threads each dispatch's outgoing headers into state.onResponse (new optional third arg) at all four fire sites (routed/rewrite × first/retry); outgoingHeadersOf() normalizes string+init and Request-object inputs.
  2. incremental-client adds the x-bili-turn-id header, a newIncrementalTurnId() generator, and a generic sweepStale().
  3. Each adapter stamps the turn id onto the outgoing request, keys its pending map by it, and attributes responses by it. The token rides the wire harmlessly (the proxy ignores unknown x-bili-* extras).

pi.ts was already immune (its ack channel carries session context) — untouched. The server lane (incremental-lane.ts) is untouched too: its per-conversation lock already makes concurrent same-URL turns safe, so the probe protocol stays 1.

Behavior change (disclosed)

  • Concurrent sessions sharing one provider URL previously fell back to full-resend while any one of them had an in-flight turn (<30 s window); now each runs its own incremental turns independently. Same-conversation concurrency stays impossible in practice (hosts send sequential generations per session; worst case the server 409s → sticky dead → resync, same class as pi).
  • The 30 s TTL now only bounds memory for network-dead planned requests — it can no longer mis-attribute. dsh-native gains a TTL where it previously had none (memory hygiene only).

Config surface

None — no new env/config/flag; the feature stays behind the existing pluginIncrementalTurns rollout gate (default off).

Tests

  • New regression tests pin the exact bug: a late same-URL response must credit the conversation whose turn id it carries, never the other one (opencode + omp suites).
  • The native-intercept suite pins per-dispatch header threading across all input shapes.
  • Local: typecheck + build clean; targeted suites green (opencode-incremental, omp-native-incremental, native-intercept, incremental-lane, native-intercept-selfheal, apig-resign, dsh-native, omp-native, opencode-native). Full-suite verdict from CI on this head.

@github-actions

github-actions Bot commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-16_fix-incr-ack-url-keying (31d0f3f)

Option A — Install this exact build (recommended)

npm install -g billion-context@0.0.4100-pr.3340.c90dca1

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr3340.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重:不重复。 open+closed issue/PR 全量检索(按症状+机制匹配):没有其他在途条目修同一个「URL-keyed pending-ack map 跨会话误归属」缺陷。相关但不同的条目:

本条是 PR #3340 的 [PR] 镜像(修 #3329,前一 agent 会话已完成实现并在 #3329 贴出八问)。按规范不重做,转入独立 review:原 issue 方向关 → 全窗 diff 分段通读 → 既有行为回归清点 → 测试有效性 → CI 状态。完成后按八问体例给合并判定。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

PR 审查结论(独立审核,非复读 issue 侧分析)

八问:

  1. 是什么问题:增量轮次功能的 pending-ack 表以请求 URL 为键;两个会话共用一个 provider URL 时,迟到的响应会把 x-bili-applied-seq 记到错误会话头上,双方水位中毒,sticky dead 标志把两个会话的增量轮次永久杀死(永久退回全量重发)。
  2. 什么原因导致的:onResponse(url, response) 观察通道没有每请求身份——URL 是唯一的关联键,而 URL 跨会话共享。修法把关联键换成请求级 x-bili-turn-id,方向正确(修的是产生缺陷的机制:关联维度缺失,不是症状层拦截)。
  3. 影响范围与严重程度:仅「并发会话共享同一 provider URL」的宿主(多 dsh 线程/进程、多 omp 会话指向上游);症状是性能退化(永久全量重发),非正确性丢失。单会话用户不受影响。功能在默认关闭的 pluginIncrementalTurns rollout 门后(percent:0)。
  4. 怎么修复的:三个适配器在出站请求上盖 turn id → native-intercept 把每个 dispatch 的实际出站头作为第三参传给 onResponse(四个 fire site 全覆盖;string+init 与 Request-object 两种输入形态、重试路径均保留已盖头)→ pending 表按 turn id 建键/清扫/归属。服务端零改动——已核实 lane 的 per-conversation lock(inFlight Set)确实让同 URL 并发安全,probe 协议保持 v1 成立。
  5. 是否有回归问题(逐项清点既有行为):
  6. 是否破坏缓存:不涉及。probe 缓存、前缀 hash、ledger 均未触碰;turn id 每请求新生成,不进入任何缓存键。
  7. 是否改了设计方向:不涉及。修复完全留在既有客户端状态机架构内,lane 协议不变。
  8. 是否引入新配置:无(已核实:无新 env/config/flag,仍在既有 rollout 门后)。

测试有效性:三个新测试断言有牙、层次正确——opencode/omp 套件钉住「迟到的同 URL 响应必须记到携带其 turn id 的会话、绝不记到另一个」(正是 #3329 的 bug field),native-intercept 套件钉住逐 dispatch 的头传递(含 Request-object 形态)。缺口:dsh-native 的 armed-flag 路径无单元测试;交错/双规划场景无测试(与上面两条残留对应,已在 #3329 清单跟踪)。

CI 状态:head 2e1e939a6 39/40 绿。唯一红 shard(ubuntu-22-2)= lineage-reconcile-boot.test.ts AC12 断言失败——与本 PR 文件零相关的既有 flake(debounce 竞态),master 已由 6f780f093(#3345)修复;分支落后 master 9 个提交,rebase 后应全绿(master 侧改动只碰 3 个测试文件,与本 PR 8 个文件无重叠,rebase 预期干净)。

审查中发现的新问题处置:

判定:rebase 到当前 master 后可合(同时清掉唯一红 shard)。按 §7.4 不可自动合并(跨 5 个模块、并发语义行为变更、guard 移除属回退行为变更)——请人工合并。

@rahul05ranjan rahul05ranjan left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I filed #3329, so I went through this against the exact sequence described there rather than just the diff summary — this fixes it, and cleanly.

What I verified:

  • Attribution mechanism. onResponse now receives this dispatch's outgoing headers as a third argument at all four send sites in native-intercept (first/retry × routed/rewrite), and each adapter keys its pending map by a fresh x-bili-turn-id stamped at plan time. I checked the runtime semantics outgoingHeadersOf() depends on with a standalone Node script against a local server: init.headers does win over a Request-object input's own headers, pre-stamped Request-object headers do ride the wire, and Headers iteration yields lowercase names — so the lowercase-record mirror matches what fetch actually sends and the headers?.[INCREMENTAL_TURN_ID_HEADER] lookup can't miss on casing.
  • The #3329 sequence, old vs new. Old: A plans at t0 (URL slot → A), B plans at t+31s (A's entry aged out, slot now holds B), A's late response lands at t+32s with x-bili-applied-seq → incPending.get(url) returns B's entry → B's watermark advances on A's seq. New: A's response carries A's own turn id, whose entry was already swept → lookup misses → nothing is attributed; B seeds untouched and A reseeds honestly. The regression tests in the opencode/omp suites pin exactly this (one shared URL, 31s clock jump), including that B later rides its own seq of 7.
  • Retry restamps. withHeaders merges rather than replaces, so a turn id stamped on the first pass survives the retry restamp even after dsh's incTurnIdArmed flag is consumed — a retried dispatch still correlates back to its pending entry.
  • Transition. The pending maps are in-process only and the shared helper state machine (noteIncrementalAck, keyed by conversation id) is unchanged, so there's no in-flight migration concern. pi.ts staying untouched is right — its ack channel already carries session context.

Two small things:

  • dsh-native.ts: sweepStale runs after the probeIncrementalTurns bail-out, so once the probe is off the pending map isn't swept until the next planned turn. Harmless now that the TTL is memory-hygiene-only, but omp and opencode sweep before the probe check — the placement differs across adapters for no obvious reason.
  • Confirming I read the omp behavior change right: dropping the single-in-flight-per-URL guard means a same-session double-dispatch (hosts shouldn't, but if one did) now plans two turns against one conversation state instead of falling back to full resend. The 409 → sticky-dead → resync backstop covers it, just noting it's a real behavior delta beyond the concurrent-sessions win.

One caveat: I couldn't run the new suites on my machine (no dev deps available here), so the test-level claims are from reading them; everything above I either traced in the code or ran directly.

@ranxianglei
ranxianglei force-pushed the 2026-10-16_fix-incr-ack-url-keying branch from 2e1e939 to a496227 Compare October 16, 2026 13:33
@ranxianglei
ranxianglei force-pushed the 2026-10-16_fix-incr-ack-url-keying branch from a496227 to ae50b02 Compare October 17, 2026 02:26
…3329)

The pending-ack maps in opencode-native, omp-native and dsh-native were
keyed by request URL. Two conversations sharing one provider URL could
cross-attribute x-bili-applied-seq when a response arrived late or the
slot was evicted, poisoning both watermarks and killing incremental turns
for both via the sticky dead flag.

- native-intercept: thread each dispatch's outgoing headers into
  state.onResponse (new third arg); outgoingHeadersOf() covers all four
  fire sites (routed/rewrite x first/retry).
- incremental-client: x-bili-turn-id header, newIncrementalTurnId(),
  generic sweepStale().
- opencode/omp/dsh adapters: stamp the turn id on the outgoing request,
  key the pending map by it, attribute responses by it; drop the
  single-in-flight-per-URL guard (concurrent conversations now run their
  own turns - the server lane's per-conversation lock already makes that
  safe); TTL becomes memory hygiene only. dsh gains a TTL (it had none).
- regression tests: same-URL late-response cross-attribution for opencode
  and omp; per-dispatch header threading in native-intercept.
@ranxianglei
ranxianglei force-pushed the 2026-10-16_fix-incr-ack-url-keying branch from ae50b02 to 31d0f3f Compare October 17, 2026 02:36
@ranxianglei

Copy link
Copy Markdown
Owner Author

Rebased onto latest master (fa41483 → 31d0f3f) — this absorbs the three follow-ups that landed since the last head:

Conflict resolution was mechanical (parallel import lines). Full CI green on the new head, all e2e lanes green; incremental suites pass locally (omp 4/4, opencode 6/6, native-intercept 59/59, dsh #3370 atomic-conversion test ✔; remaining local dsh-native failures are the known sandbox env-class set, byte-identical failure list on clean master). Ready to merge.

@ranxianglei
ranxianglei merged commit f054c9d into master Oct 17, 2026
44 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

📦 Merged Build Published

This merge to master was published as 0.0.514-master.f054c9d (rolling master tag).

npm install -g billion-context@0.0.514-master.f054c9d  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants