Skip to content

Incremental turns phase 2: marker divert on ordinary model requests + native client helpers (#3248) - #3273

Merged
ranxianglei merged 5 commits into
masterfrom
2026-10-15_incremental-native
Oct 15, 2026
Merged

ranxianglei merged 5 commits into
masterfrom
2026-10-15_incremental-native

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

Phase 2 of #3248: a native extension sends incremental turns without changing its provider URL or calling a dedicated endpoint — it stamps x-bili-plugin-turn: 1 on its ORDINARY model request and posts the incremental payload there.

Why this shape

  • The URL never changes. Hosts keep whatever provider configuration they have; the divert is content-negotiated, not routed.
  • The body shape is the final word. Top-level conversationId + sinceSeq + messages and NO top-level model = incremental; anything else (a normal anthropic body) rides the normal pipeline unchanged even with the marker — a host may stamp the marker whenever the feature is enabled and convert only the turns it wants, and a raced-off probe can never break a request.
  • One marker + one body + one probe, identical for every native host (pi, dsh-native, opencode, omp-native, future ones): the shared client state machine lives in src/agent/incremental-client.ts (probe TTL cache, wire-hash prefix diffing, sticky system, per-conversation dead flag, ack bookkeeping). Reference wirings: pi (extension events) and dsh-native (fetch patch); the others follow the same in-code recipe.

Rollout (requirement: default = zero impact)

  • Rollout feature pluginIncrementalTurns, default 0% — every install stays byte-identical until enabled. Force on with rollout.features.pluginIncrementalTurns: true (or a percent) or BILI_ROLLOUT_PLUGIN_INCREMENTAL_TURNS=1.
  • Still requires a live unified sidecar (shadow/full); legacy-only lanes report storage: "legacy-only" on the probe and answer 503 on shaped turns only — plain traffic is untouched.
  • Capability probe: GET /__bili/plugin/turn → { ok, protocol: 1, feature, enabled, storage } (loopback-gated).

Semantics carried over from phase 1 (+ one addition)

  • Seeds imply resync: true — the first incremental turn never 409s, whether the canonical is empty (fresh) or already live (interop with full-resend traffic: the client's view is authoritative).
  • NEW: a system-only change (same message set, different system prompt) rides an empty delta — canonical unchanged, sticky system updated server-side, request forwarded. A same-turn retry still rides the normal path (skip).

Server changes

  • incrementalTurnBodyShape() strict shape check; marker divert in handle() buffers the body once, diverts shaped turns to the phase-1 lane (forwarding to the ORIGINAL path), and replays non-shaped ones through handleRequest with the marker stripped (same admission, no loop).
  • applyTurn accepts an empty delta when a system update is present.

Tests

tests/incremental-lane.test.ts 15/15: probe rollout states (default off / env force on), marker divert e2e (seed + delta assembled upstream, x-bili-applied-seq), non-incremental ride (model rides, no divert artifacts, text preserved through the normal pipeline), disabled rollout (503 shaped / 200 plain), helper state machine (seed-implies-resync, delta slicing, edit → resync, sticky system incl. empty-delta system update, dead flag), body-shape unit (accepts incremental, rejects anthropic/garbage/bad seq).

Regressions: dsh-fork-adopt, issue833 plugin lane, #1542, plugin-fork, side-lane-invariants, #2601 all green; tsc clean; unused-export gate clean (4 waived). dsh-native.test.ts has the known local 15s-timeout env family (reproduces on clean master; CI green).

Fixes the native-access half of #3248 (the dsh client adoption and the server-authored prefix fast path remain on the issue).

Native extensions send incremental turns on their ordinary model requests:
stamp x-bili-plugin-turn: 1 alongside the usual x-bili-plugin header and post
the incremental payload — the body shape (conversationId/sinceSeq/messages,
no top-level model) is the final word, so a marker on a non-incremental body
replays through the normal pipeline byte-semantically unchanged, and hosts
keep their provider URLs untouched. The lane forwards to the ORIGINAL path.

- rollout feature pluginIncrementalTurns (default 0% = byte-identical
  installs; force via config rollout.features.pluginIncrementalTurns or
  BILI_ROLLOUT_PLUGIN_INCREMENTAL_TURNS=1), still requires a live unified
  sidecar; disabled lanes answer 503 on shaped turns only
- GET /__bili/plugin/turn capability probe: {ok, protocol:1, feature,
  enabled, storage}
- seeds imply resync:true (first incremental turn never 409s; interop
  full-resend traffic replaced client-authoritatively); a system-only
  change rides an empty delta (canonical unchanged, sticky system updated)
- src/agent/incremental-client.ts: shared client state machine (probe TTL
  cache, wire-hash prefix diffing, sticky system, dead flag, ack bookkeeping)
- reference wirings: pi (before_provider_headers marker + payload
  replacement + after_provider_response ack) and dsh-native (fetch patch
  beforeSend/headersFor/onResponse; opencode/omp-native follow the same
  recipe per the in-code notes)

Tests: probe rollout states, marker divert e2e (seed+delta assembled),
non-incremental ride, disabled 503 + plain ride, helper state machine
(seed/delta/edit-resync/sticky-system-empty-delta/dead), body-shape unit.
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重:不重复。 按失败签名/主题检索 open+closed 的 issue 与 PR(x-bili-plugin-turn、incremental native 等):相关存量仅父 issue #3248(已关闭)、#3253(阶段 2 端点本体,已合)、#3257(阶段 1 reconcile 快路径,在途,机制不同)。本 PR 是 #3248「native-access 半边」的后续实现,与三者均非同机制重复项。

处理姿态:这是 [PR] 工单——拉分支核对代码、确认 CI 状态、按八问体例报结论,不重做。正在审:单 commit 13c1ade4f,10 文件 +697/-14(server divert + 新共享客户端状态机 + pi/dsh-native 接线 + rollout 门控)。初步已注意到一个待核点:新增 env BILI_ROLLOUT_PLUGIN_INCREMENTAL_TURNS 似乎没进 CONFIGURATION.md(在途的 #3256 台账守卫会拦这类未登记 BILI_*,需确认是否豁免或补文档)。稍后给完整结论。

@github-actions

github-actions Bot commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-15_incremental-native (44d917c)

Option A — Install this exact build (recommended)

npm install -g billion-context@0.0.3777-pr.3273.7a73e6b

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr3273.tgz
npm install -g package

Install instructions are refreshed on each push.

ranxianglei and others added 4 commits October 15, 2026 17:24
…NAL_ONLY exports

- scripts/check-unused-exports.mjs: git ls-files --cached --others
  --exclude-standard — a brand-new module is exactly where INTERNAL_ONLY
  leaks happen, and a tracked-only scan passed locally right before CI
  failed on the same tree (the fresh src/agent/incremental-client.ts was
  untracked at the local gate run on #3273).
- src/agent/incremental-client.ts: IncrementalPlan / PlanArgs keep their
  leading export off (module-internal shapes; callers infer).
… instead of replaying an empty body (#3273 review)

A >cap (or stream-erroring) body on a marker'd request used to be
swallowed: readBody rejects AND destroys the socket, the catch left an
empty buffer behind, and the replay path re-sent that empty buffer into
the normal pipeline — an upstream-facing 400/reset where the identical
unmarked request gets bili's own 413 JSON. Mirror the pipeline's own
body-read failure classification (413 on BodyTooLargeError, 400 on any
other stream error) and stop.
…ne (#3273 review)

The phase-2 addition (applyTurn accepts an empty delta when a system
update is present) was only exercised at the client-helper level. Add
the end-to-end pin: shaped turn with messages=[] + new system forwards
the assembled request with the updated sticky system, canonical length
unchanged; an empty delta WITHOUT a system stays a 400.
…sions (#3273)

withFeatures merges fixtures into the live ROLLOUT_FEATURES registry and
restores only what it added itself; the exact-list assertion predates any
shipped entry (#3062 shipped the registry empty). The pluginIncrementalTurns
entry added by this branch made the assertion fail deterministically on
every shard-3 CI lane. Assert the sorted union of shipped + fixture keys
and keep the per-feature enabled/source checks sharp.
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Review 完成。我没有重做工作,而是拉分支逐文件核对了全部 10 个文件的 diff、跑了定向验证、复现并修掉了 CI 红。结论:可合并——新 head 44d917ca7 全绿(30/30 checks),另有一个 409 恢复语义的决策项等你拍板(见文末)。

结构化八问

  1. 是什么问题:功能 PR(Delta lane: native-mode clients send only deltas, server maintains full state + one-time fold, upstream receives snapsho | 增量车道:native 模式客户端只发增量,服务端全量维护+一次折叠,上游只收快照(#2926 的泛化) #3248 phase 2 原生接入半边)——native 扩展不改 provider URL、不调专用端点,只在普通模型请求上盖 x-bili-plugin-turn: 1;body 形状匹配(顶层 conversationId+sinceSeq+messages、无顶层 model)就 divert 到 phase-1 lane,否则剥掉 marker 走正常管线重放。外加共享客户端状态机(src/agent/incremental-client.ts)、pi/dsh-native 两处接线、0% 默认 rollout 门。
  2. 什么原因导致的:不适用(非 bugfix)。但 review 发现 3 个真实问题,已直接修为分支上独立标注 commit(见第 4 问)。
  3. 影响范围与严重程度:默认零影响——rollout 默认 0%,且两处接线都先 probe 确认 enabled 才盖 marker,默认安装连 divert 分支都进不去(有测试钉住)。启用后只影响 pi/dsh-native 自身流量。严重程度:无(默认关)。
  4. 怎么修复的(review 发现 → 已修):
    • f9a326a02 — divert 块里 readBody 失败(超 100MiB cap 或流错误)被 catch 吞掉后空 body 重放上送,而同样的未标记请求在正常管线会拿到 bili 自己的 413 JSON(request_too_large)或 400。改为镜像管线自身的错误分类(未断 socket 时回 413/400,不断则让 socket 死亡说话)。
    • fdce810aa — 新增行为「空 delta 承载 system-only 变更」此前只有客户端 helper 层覆盖,补了服务端 e2e 钉住(seed SYSTEM-A → 空 delta+SYSTEM-B → canonical 不变、上游看到新 system;无 system 的空 delta → 400)。incremental-lane 现 16/16。
    • 44d917ca7 — 真回归:新增 registry 条目 pluginIncrementalTurns 打破了 tests/rollout.test.ts:196 的精确列表断言(该测试写于 registry 为空的时代,helper 是合并语义不是替换),导致 CI shard-3 四个 OS×node lane 确定性红。改为「shipped ∪ fixture 排序并集」断言,per-feature(enabled/source)检查保持锐利。本地 25/25。
  5. 是否有回归问题:逐项过过既有行为——默认安装(不盖 marker→永不进 divert)、非 shaped 标记 body 剥 marker 正常重放(有测试)、GET probe loopback-gated(adminGateDenial)、POST turn 路由不变、/__bili/* 排除在 divert 之外、marker 剥离保证下游字节等价——均保持。配置面合规:registry 条目是 feat: 灰度发布机制 —— 新特性按安装维度百分比放量(支持嵌套灰度 A→B) #3062 机制自己的采纳路径,BILI_ROLLOUT_<FEATURE> 是 CONFIGURATION.md en+zh 已文档化的通用约定,[PR] Stale item cleanup: Document transitional feature flag registry + CI guard (BILI_* outside registry → immediate red) | [PR] 陈年件清理:过渡开关台账进文档 + CI 守卫(台账外 BILI_* 直接红) #3256 的 ledger guard 明确豁免动态 token,无交互问题。无回归。另有两族 flake(claude-native port-zone 漂移 ×3、pi-subagents maxConcurrent 时序 ×1)在本机并发负载下复现,但 master 与 head 单独跑均通过→环境 flake,与本 diff 无关(本 diff 不触碰那些路径);新 head 全量 CI 亦未复现。
  6. 是否破坏缓存:不涉及——组装后的请求完整重入正常管线(session/fold/nudge/tools 全按原路径走),marker 在重放前剥除,前缀缓存/fold ledger/重试策略不受影响。
  7. 是否改了设计方向:没有——严格按你 floor 9 宣布的形状执行(URL 永不变、body 形状是最终裁决、一套 marker+状态机供所有 native host 复用);剩余项(dsh client adoption、server-authored prefix fast path、SSE 聚合)留在 Delta lane: native-mode clients send only deltas, server maintains full state + one-time fold, upstream receives snapsho | 增量车道:native 模式客户端只发增量,服务端全量维护+一次折叠,上游只收快照(#2926 的泛化) #3248。一个语义分歧需要你定(见下)。
  8. 是否引入新配置:是——rollout.features.pluginIncrementalTurns(registry 条目,默认 percent 0)+ 通用 env BILI_ROLLOUT_PLUGIN_INCREMENTAL_TURNS(1/0 强制开/关)。无需迁移:默认关 = 现状字节级一致。

待拍板:409 恢复语义(docs 与实现不一致)

  • 现状:PLUGIN.md:203 承诺「409 … client MUST recover with a resync turn」;但实现的接线(pi.ts:1337、dsh-native.ts:1392)对 409 直接 markIncrementalDead = sticky 回落 full-resend 直到进程重启;planIncrementalTurn(forceResync) 参数存在却从未被调用;模块头注释("for that turn")与 markIncrementalDead 注释("Sticky per-conversation")自相矛盾。另外 503(lane 级条件)也设了 dead 同时清 probe「下轮重探」——dead 使该会话的重探永远无效,sidecar 恢复后其他会话能回来、唯独见过 503 的会话不能。
  • 选项 A(推荐,最小):保留 409→sticky-dead;把 PLUGIN.md 措辞改成与实际一致(回落 full-resend,canonical 经普通流量自然再收敛);去掉 503 处的 dead 设置(只留 clearIncrementalProbe,让重探真正生效);修模块头注释。影响面:文档 + ~5 行行为;回归风险:≈0(实际行为比文档更保守,A 只是把谎话改真)。
  • 选项 B:实现 409 一次性 resync 恢复(409 时置 resyncNext 标志,下一发请求 forceResync 重规划;第二次 409 才 dead)。影响面:helper + 两处接线 + 测试 ≈30 行;好处:长寿命进程内一次瞬态 409 不再永久失去优化(主要 409 来源正是 persist 与 ack 之间崩溃,恰是 resync 便宜可恢复的场景);回归风险:在 PLUGIN.md 声明的 single-writer-per-conversation 假设下安全,但若两个进程共享同一 conversation id,stale 视图的 resync 会覆盖更新的 canonical——该隐患今天 seed 就已存在,B 不引入新的风险类别,只提高触发频率。
  • 我的建议:A 先行(小,消除误导契约 + 修 503 瑕疵),B 作为 follow-up;也可以一起做。你拍板后我直接推标注 commit。

CI 轨迹(备查)

head 结果
13c1ade4f unused-exports 红(作者已在 356729fdd 修复)
356729fdd shard-3 ×4 红 + win22 cancelled(rollout 测试回归,即 44d917ca7 所修)
fdce810aa(我的前两个 commit) shard-3 ×4 红(同上)+ static-gates win22 红(pi-subagents 时序 flake,日志实锤,单跑通过)
44d917ca7 30/30 全绿

判定:可合并(默认关、零影响、CI 全绿、review 发现均已闭环)。409 语义项不阻塞合并,拍板后我按选项落地。

@ranxianglei
ranxianglei merged commit 0e9360a into master Oct 15, 2026
41 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

📦 Merged Build Published

This merge to master was published as 0.0.448-master.0e9360a (rolling master tag).

npm install -g billion-context@0.0.448-master.0e9360a  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

ranxianglei added a commit to sorinirimies/billion-context that referenced this pull request Oct 15, 2026
master moved past the PR base (ranxianglei#3273 incremental lane, ranxianglei#3199 unified web
lists, codexForge kind=prompt, network.allowPrivateHttpProviders docs).
Conflicts were all additive:

- src/server.ts: keep master's codexForge kind union (+"prompt") AND the
  PR's autoModeForge Prepared field.
- src/server/handle.ts: combined import lines (knobs + auto-mode-forge +
  v1internal); PR interception point verified to still run BEFORE the
  ranxianglei#388 side-request split.
- CONFIGURATION.md / CONFIGURATION.zh-CN.md: keep both new env-table rows
  (BILI_ALLOW_PRIVATE_HTTP_PROVIDERS, BILI_AUTO_MODE_FORGE), alphabetical.

Verified: tsc clean, 56/56 (auto-mode-forge + e2e-anthropic + web-i18n +
text-safe), test-suite-lock gate, import-graph gate, build OK.
@github-actions github-actions Bot mentioned this pull request Oct 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant