Repository navigation
[PR] fix(dsh): size each llm/stream dispatch against its own model and await a cold window - #7066
xiaobaifly7 wants to merge 1 commit into
Conversation
…it a cold window headersFor stamped x-bili-plugin-model / -context-window for the model the session header names (ranxianglei#2381), but an llm/stream dispatch can stream to a different model: auxiliary calls (compaction, title, subagent) running on another model carried the session model's numbers, so the proxy's exact-match gate discarded the header channel and the request was sized against the fallback window. The window resolve was also fire-and-forget, so a model's first dispatch always went out with no window header. The dispatch scope now carries the model the dispatch streams to (GenerateOptions.provider + model, falling back to the session header and the global default). A dispatch whose model has no cached window waits for the in-flight resolve before its first fetch, bounded to 2s from the resolve's start so a slow or hung resolver can never stall the stream. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b) 查重: 不重复 — 已核对本仓全部 open [PR] 行,没有第二个 PR 针对本线程(#7065 / dsh dispatch model-window);本 PR 是该线程唯一在途修复。 标签: bug + dsh + scope/wiring + p2(已加,保留 pr) — 缺陷类;仅 dsh 车道;客户端接入层(dsh-native 适配器);可用但有明显问题(辅助 dispatch 按 fallback 窗口定尺寸、首派无 window 头),非 p1 类(不致不可用、非缓存泄露)。 分支已拉取核对: |
[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b) PR Review 结论:可合(待 CI 绿,人工合并)外部贡献者 PR,质量过 ⑤b 硬标准(命名达意、无死代码、错误处理完整、常量具名、注释讲为什么、无新依赖),测试是真测试有牙(负向控制独立复核成立),方向打到了根因。不需要对齐重构。 结构化八问
行为变更清单(旧 → 新 + 原因)
本地定向验证(分支
|
[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b) CI 状态更新(实时核对):head
最终判定不变:可合,等人工合并。 剩余关注点只有一条:#6123 仍 open 且动同一热区( |
(ai: anthropic/claude-opus-5-5)
Branch:
2026-11-17_dsh-dispatch-model-window(one commit on77fe0b1d)Problem
headersFor(window/model/max-output headers) andeffectiveContextWindow(oversize arm of the takeover gate) resolve the target model as session header → global default (#2381) and read a cache thatrefreshModelInfofills fire-and-forget. So:llm/streamdispatch that streams to a different model than the session header (compaction, session title, subagent) is stamped with the session model's numbers. The proxy honors the header window only when the header model matches the body model, so the channel is discarded and the request is sized against the fallback. The auxiliary model is never resolved or reported.x-bili-plugin-context-window, because the resolve it kicked off lands after the synchronous stamp.Root cause
The model target is derived from ambient session state rather than from the dispatch itself, even though the dispatch's own options (the same
llm/streamoptions #4082/#4491/#6880 already read) name the model it streams to. The resolver's result is never awaited by anything on the request path.Fix (
src/agent/dsh-native.ts, dsh lane only)LlmDispatchScopegainsmodelTarget. Thellm/streamhandler fills it fromoptions.{provider,model}(both are required fields of dsh'sGenerateOptionsin 0.2.2-rc.3), else the session header / global default (unchanged fallback chain), andsessionModelTargetOfreturns the scoped target first.headersFor,effectiveContextWindowanddeclaredWindowForDirectall go throughsessionModelTargetOf, so they follow without further changes.refreshModelInforeturns a wait handle for the key's in-flight resolve, orundefinedwhen there is nothing to wait for (cached window, retry cooldown, no services/resolver).refreshingKeysbecomes aMapholding that handle so concurrent dispatches share one resolve. The resolve itself, the per-key commit, the 130k上下文窗口被占满触发无效压缩越压越大 #1812 cooldown and the runtime-info report are untouched.next()behind it:async function* () { await pending; yield* next(); }. The deferred call still runs insiderunLlmDispatch/wrapScopedIterable, so Bili skips compression in the DSH Remote scenario, with local fix and test attached | Bili 在 DSH Remote 场景下跳过压缩,附本机修复与测试 #5614's boundary and the sid/msgIds/purpose scope are unchanged. Cache hits take the old eagernext()path.Promise.race([resolve, 2s timer from resolve start])(timerunref'd). A slow or hung resolver delays only the dispatches that arrive within 2s of the resolve starting, and then the request goes out exactly as before the fix (no window header). A resolve that is merely slow still lands in the cache for later requests.Scope / not changed
sessionIdstill pass through untouched (documented Why did it spike to several M tokens: after dsh attribution was lost, requests bypassed bili and were sent directly, ren | 为什么会涨到几 M tokens:dsh 归因丢失后请求绕过 bili 直发,压缩与护栏全部失效 #4082 contract: the agentless auto-review lane keeps its existing attribution ladder). Scoping them would also touchpfCaptureIds, the Bili skips compression in the DSH Remote scenario, with local fix and test attached | Bili 在 DSH Remote 场景下跳过压缩,附本机修复与测试 #5614 boundary and msgIds capture. Out of scope here.Tests
New
tests/dsh-dispatch-model-window.test.tsdrives the realapply()+llm/streamwaterfall against a mock attach proxy:provider,model: "aux-model") stampsaux-modeland its window on its first request, while a dispatch on the session's own model keeps that model's window;Negative control (verified locally): with only
src/agent/dsh-native.tsreverted to77fe0b1d, all three fail (actual: 'main-model'vsexpected: 'aux-model';actual: undefinedvsexpected: 'fresh-model'; the hung case stamps the session model's window). With the fix all three pass.Local targeted runs on the branch:
npm run typecheckclean;dsh-dispatch-model-window3/3,dsh-llm-stream-attribution7/7,dsh-msgids6/6,issue5614-dsh-remote-fetch-boundary3/3,issue4894-direct-window-guard5/5,issue1569-window-stamp-and-anchor8/8,dsh-native75/76. The singledsh-nativefailure (#3578 transition contract …) is a WindowsEPERMon temp-dir cleanup and fails identically on an untouched77fe0b1dcheckout. Full-suite verdict: CI.Behavior change
Review notes
llm/streamhandler signature andLlmDispatchScope. Expect a textual conflict, not a semantic one.Fixes #7065
🤖 Generated with Claude Code