Skip to content

[PR] fix(dsh): size each llm/stream dispatch against its own model and await a cold window - #7066

Open
xiaobaifly7 wants to merge 1 commit into
ranxianglei:masterfrom
xiaobaifly7:2026-11-17_dsh-dispatch-model-window
Open

xiaobaifly7 wants to merge 1 commit into
ranxianglei:masterfrom
xiaobaifly7:2026-11-17_dsh-dispatch-model-window

Conversation

@xiaobaifly7

Copy link
Copy Markdown

(ai: anthropic/claude-opus-5-5)

Branch: 2026-11-17_dsh-dispatch-model-window (one commit on 77fe0b1d)

Problem

headersFor (window/model/max-output headers) and effectiveContextWindow (oversize arm of the takeover gate) resolve the target model as session header → global default (#2381) and read a cache that refreshModelInfo fills fire-and-forget. So:

  1. An llm/stream dispatch that streams to a different model than the session header (compaction, session title, subagent) is stamped with the session model's numbers. The proxy honors the header window only when the header model matches the body model, so the channel is discarded and the request is sized against the fallback. The auxiliary model is never resolved or reported.
  2. A model's first dispatch always goes out without x-bili-plugin-context-window, because the resolve it kicked off lands after the synchronous stamp.

Root cause

The model target is derived from ambient session state rather than from the dispatch itself, even though the dispatch's own options (the same llm/stream options #4082/#4491/#6880 already read) name the model it streams to. The resolver's result is never awaited by anything on the request path.

Fix (src/agent/dsh-native.ts, dsh lane only)

  • LlmDispatchScope gains modelTarget. The llm/stream handler fills it from options.{provider,model} (both are required fields of dsh's GenerateOptions in 0.2.2-rc.3), else the session header / global default (unchanged fallback chain), and sessionModelTargetOf returns the scoped target first. headersFor, effectiveContextWindow and declaredWindowForDirect all go through sessionModelTargetOf, so they follow without further changes.
  • refreshModelInfo returns a wait handle for the key's in-flight resolve, or undefined when there is nothing to wait for (cached window, retry cooldown, no services/resolver). refreshingKeys becomes a Map holding that handle so concurrent dispatches share one resolve. The resolve itself, the per-key commit, the 130k上下文窗口被占满触发无效压缩越压越大 #1812 cooldown and the runtime-info report are untouched.
  • When a handle is returned, the handler defers next() behind it: async function* () { await pending; yield* next(); }. The deferred call still runs inside runLlmDispatch/wrapScopedIterable, so Bili skips compression in the DSH Remote scenario, with local fix and test attached | Bili 在 DSH Remote 场景下跳过压缩,附本机修复与测试 #5614's boundary and the sid/msgIds/purpose scope are unchanged. Cache hits take the old eager next() path.
  • The handle is Promise.race([resolve, 2s timer from resolve start]) (timer unref'd). A slow or hung resolver delays only the dispatches that arrive within 2s of the resolve starting, and then the request goes out exactly as before the fix (no window header). A resolve that is merely slow still lands in the cache for later requests.

Scope / not changed

Tests

New tests/dsh-dispatch-model-window.test.ts drives the real apply() + llm/stream waterfall against a mock attach proxy:

  1. an auxiliary dispatch on another model (provider, model: "aux-model") stamps aux-model and its window on its first request, while a dispatch on the session's own model keeps that model's window;
  2. a session on a not-yet-resolved model stamps its window on the first dispatch;
  3. a resolver that never settles delays the first dispatch by about the bound, and a later dispatch for the same model does not wait again.

Negative control (verified locally): with only src/agent/dsh-native.ts reverted to 77fe0b1d, all three fail (actual: 'main-model' vs expected: 'aux-model'; actual: undefined vs expected: 'fresh-model'; the hung case stamps the session model's window). With the fix all three pass.

Local targeted runs on the branch: npm run typecheck clean; dsh-dispatch-model-window 3/3, dsh-llm-stream-attribution 7/7, dsh-msgids 6/6, issue5614-dsh-remote-fetch-boundary 3/3, issue4894-direct-window-guard 5/5, issue1569-window-stamp-and-anchor 8/8, dsh-native 75/76. The single dsh-native failure (#3578 transition contract …) is a Windows EPERM on temp-dir cleanup and fails identically on an untouched 77fe0b1d checkout. Full-suite verdict: CI.

Behavior change

  • Auxiliary dispatches on another model now carry that model's model/window/max-output headers, and the model gets resolved and reported.
  • A model's first dispatch may start up to 2s later (only when its window is not cached yet and its resolve is still pending). This new bound is a timing constant. Flagging it for human review per §7.4 rather than auto-merge.

Review notes

Fixes #7065

🤖 Generated with Claude Code

…it a cold window

headersFor stamped x-bili-plugin-model / -context-window for the model the
session header names (ranxianglei#2381), but an llm/stream dispatch can stream to a
different model: auxiliary calls (compaction, title, subagent) running on
another model carried the session model's numbers, so the proxy's exact-match
gate discarded the header channel and the request was sized against the
fallback window. The window resolve was also fire-and-forget, so a model's
first dispatch always went out with no window header.

The dispatch scope now carries the model the dispatch streams to
(GenerateOptions.provider + model, falling back to the session header
and the global default). A dispatch whose model has no cached window waits for
the in-flight resolve before its first fetch, bounded to 2s from the resolve's
start so a slow or hung resolver can never stall the stream.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

查重: 不重复 — 已核对本仓全部 open [PR] 行,没有第二个 PR 针对本线程(#7065 / dsh dispatch model-window);本 PR 是该线程唯一在途修复。

标签: bug + dsh + scope/wiring + p2(已加,保留 pr) — 缺陷类;仅 dsh 车道;客户端接入层(dsh-native 适配器);可用但有明显问题(辅助 dispatch 按 fallback 窗口定尺寸、首派无 window 头),非 p1 类(不致不可用、非缓存泄露)。

分支已拉取核对:ffbcdbf1 为 77fe0b1d(当前 master)之上单 commit,与 master 无冲突(mergeable=true, blocked 仅因 CI/审批)。正在做完整 review(原 issue 方向 → 回归清点 → 代码 → 测试有效性 → CI),结论按八问体例随后补上。

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

PR Review 结论:可合(待 CI 绿,人工合并)

外部贡献者 PR,质量过 ⑤b 硬标准(命名达意、无死代码、错误处理完整、常量具名、注释讲为什么、无新依赖),测试是真测试有牙(负向控制独立复核成立),方向打到了根因。不需要对齐重构。

结构化八问

  1. 是什么问题:dsh 原生插件给 llm/stream dispatch 打 model/window/max-output 头时,一直用「会话头模型」的数值,且窗口缓存是 fire-and-forget 填充——(a) 辅助 dispatch(compaction/title/subagent)实际流到另一个模型时被打上会话模型的数值,proxy 的 exact-match 门(pluginHeadersMatchModel)把整个头通道丢弃,请求按 fallback 定尺寸,辅助模型也从未被 resolve/上报;(b) 任何模型的首个 dispatch 必然无 window 头(resolve 还没落地)。
  2. 什么原因导致的:model target 从 ambient 会话状态(会话头 → 全局默认)推导,而不是从 dispatch 自身推导——尽管 dispatch 自己的 options 就带着它要流的模型;且 refreshModelInfo 的 resolve 结果在请求路径上没有任何人 await。
  3. 影响范围与严重程度:仅 dsh 插件车道。触发条件:辅助 dispatch 用与会话头不同的模型,或会话模型 ≠ 启动时全局默认/中途切换。严重度 p2——请求仍可用(proxy 降级到 configured/registry/default fallback 窗口),但窗口定尺寸错误、辅助模型永远不进 runtime-info 表、首派必然无 window 头。
  4. 怎么修复的:(a) dispatch scope 增加 modelTarget:llm/stream handler 从 options.{provider,model} 解析,miss 时回退会话头/全局默认(原链不变),sessionModelTargetOf 优先 scoped target,headersFor/effectiveContextWindow/declaredWindowForDirect 自动跟随;(b) refreshModelInfo 返回该 key in-flight resolve 的 wait handle(并发 dispatch 共享同一 handle,refreshingKeys Set→Map),handler 拿到 handle 时把 next() 延迟到它之后,等待以 Promise.race([flight, 2s unref timer]) 封顶;超时回退到修复前行为(无头发出),单纯慢的 resolve 仍落缓存供后续请求。
  5. 是否有回归问题:逐项清点过——无 sessionId 的 dispatch 路径原封(eager next(),Why did it spike to several M tokens: after dsh attribution was lost, requests bypassed bili and were sent directly, ren | 为什么会涨到几 M tokens:dsh 归因丢失后请求绕过 bili 直发,压缩与护栏全部失效 #4082 契约保持);缓存命中的 dispatch pending === undefined,next() 仍 eager 调用,与旧路径逐字节一致;130k上下文窗口被占满触发无效压缩越压越大 #1812 重试冷却、per-key commit、runtime-info 上报逻辑未动;Bili skips compression in the DSH Remote scenario, with local fix and test attached | Bili 在 DSH Remote 场景下跳过压缩,附本机修复与测试 #5614 边界保持(延迟的 next() 仍在 runLlmDispatch/wrapScopedIterable 的 scope 内、首个 iterator advance 时调用,fetch-chain 重入清除语义不变);refreshingKeys Set→Map 语义等价(含测试重置的 .clear());resolve 失败路径 flight 链自带 .catch,wait handle 永不 reject,await 不会炸流——未引入新失败模式。两处小漂移(见下方行为变更清单第 3、4 条),均判定安全。
  6. 是否破坏缓存:否。modelInfo 缓存 key 结构(${provider}\u0000${model})、容量 32、驱逐策略未动;等待机制只读不写缓存;wire/body 零改动,还是同样三个头只是值正确,prefix cache / pf 水位通道不受影响。
  7. 是否改了设计方向:否。dsh 车道内单模块修正,方向是 DSH插件按「全局默认模型」上报上下文窗口,导致会话真实模型匹配不到窗口、回落默认200K #2381「per-session target」向「per-dispatch target」的自然延伸,proxy 侧零改动。
  8. 是否引入了新的配置:否。无新 env/config/CLI;MODEL_INFO_DISPATCH_WAIT_MS = 2000 是内部时序常量,非用户面 knob。

行为变更清单(旧 → 新 + 原因)

  1. 异模型辅助 dispatch:旧 = 打会话模型数值(头通道被 exact-match 门丢弃)→ 新 = 打自身模型数值,且该模型被 resolve 并上报。原因:修复本身。
  2. 冷模型首派:旧 = 立即发出、无 window 头 → 新 = 最多等 2s in-flight resolve 后发出,超时回退旧行为。原因:新增时序常量 2000ms,PR 正文已按 §7.4 提请人工 review。
  3. (小)冷 case 的流启动时机:旧 = handler 返回时同步调 next() → 新 = 首个 iterator advance 时调(最晚 2s 后)。原因:等待机制的必要推论;宿主对 AsyncIterable 的消费语义不变,scope/sid/msgIds 在 handler 时已捕获进闭包。
  4. (小)嵌套 dispatch(无显式 model)的 fallback:旧 = 会话头模型 → 新 = 外层 dispatch 的模型。原因:scoped target 优先;边缘 case,dsh GenerateOptions 声明 provider+model 为必填,实际难触发。

本地定向验证(分支 ffbcdbf1,非全量)

  • tests/dsh-dispatch-model-window.test.ts:3/3 过;负向控制独立复核——只把 src/agent/dsh-native.ts 回退到 77fe0b1d,3/3 全挂,恢复后全过(测试有效性成立,不是假测试)。
  • 邻域套件 dsh-llm-stream-attribution / dsh-msgids / issue5614-dsh-remote-fetch-boundary / issue4894-direct-window-guard / issue1569-window-stamp-and-anchor:29/29 过。
  • dsh-native:76/76 过(作者报的 75/76 中那 1 个是 Windows EPERM 临时目录清理环境问题,Linux 上不复现)。
  • npm run typecheck:clean。
  • CI:review 时 head ffbcdbf1 上仍在跑(24 success / 19 skipped / 其余 in progress),最终判定以 CI 为准。

其他

  • Auto-merge 门:不符合(wire 头值变更 + 新时序常量 + fallback 行为变更)→ 人工 review/merge,PR 正文已自行声明。
  • 热文件:[PR] feat(dsh): route models between stock native and bili compaction #6123(feat(dsh): route models between stock native and bili compaction)仍 open,同样动 llm/stream handler 签名与 LlmDispatchScope——两者后合者需 rebase(文本冲突,非语义)。
  • 标签:bug + dsh + scope/wiring + p2(首条回复已加)。

判定:代码与测试均达标,可合;等 CI 绿后由人工合并。

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

CI 状态更新(实时核对):head ffbcdbf1 上 CI 已绿。

  • 原始 push run(14:42Z)完整跑了 core 矩阵并全绿:test (ubuntu 22/24 × shards, windows 22 × shards) 与 static-gates 全部 success(27 success / 1 platform-specific skip),无 fail。
  • 14:51Z 的后续 run 是 label 事件触发的 skip-scoped 重触发(ci.yml plan job 对非 ci:full 的 labeled 事件 scope=skip),以 success 收尾,不覆盖也不否定原始 run 的全量结果。
  • 之前 review 里「CI 仍在跑」的说法已过时,以此条为准。

最终判定不变:可合,等人工合并。 剩余关注点只有一条:#6123 仍 open 且动同一热区(llm/stream handler 签名 + LlmDispatchScope),两者后合者需 rebase(文本冲突,非语义)。

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

2 participants