Repository navigation
[PR] fix(#1812): recognize vLLM overflow errors — honest rescue, configuration is the sizing source - #1838
[PR] fix(#1812): recognize vLLM overflow errors — honest rescue, configuration is the sizing source#1838ranxianglei wants to merge 3 commits into
Conversation
…t to the stated window vLLM rejects oversized requests with 'prompt (N tokens) + max tokens (M) exceeds the context (W); requests are never truncated' — a phrasing without the word 'window', so inspectContextOverflow never matched it. The 400 passed through verbatim and clients that treat it as fatal locked the session in a 400 retry loop even when the payload was rescuable (#1812 log: declared window 200000 vs real 131072). Three coordinated changes: - util: detect the vLLM dialect, extract the stated window W and the stated prompt size N (new ContextOverflowInfo.promptTokens). - server/budget: the overflow-refold rebuild's output clamp now runs against min(declared, stated window) with a FRESH input estimate floored by the upstream's own prompt count. Previously it ran against the declared window while lastInputTokens was armed to the stated window (#987), which either left the client's max_tokens untouched (input alone fit) or zeroed the cap. The normal (non-refold) clamp path is byte-for-byte unchanged. - server: thread the stated window + prompt count through the refold rebuild (runPreparedPipeline -> prepare{Openai,Google,Responses} -> clamp). Tests: dialect unit cases (verbatim log message, comma grouping, sglang negative for promptTokens) and a hermetic e2e pinning the full chain on the OpenAI wire: reject -> arm -> refold clamps max_tokens -> re-send once in the same request -> 200, provably fitting the real window.
…tion is the sizing source The #1812 incident, decomposed: a relay-fronted vLLM model with a real 131072 window was sized against bili's unconfigured 200k fallback (handshake gap — dsh side fixed in #1836), so compression never fired in time; when the server finally rejected with vLLM's phrasing — 'prompt (N tokens) + max tokens (M) exceeds the context (W); requests are never truncated' — the detector did not match it (no 'window'/'context length' wording), so bili passed the 400 through verbatim and the client retried identical requests into a hard loop. This PR fixes the proxy side: - recognition: the vLLM dialect matches CONTEXT_OVERFLOW_PATTERN → arms the #1110 shrink and retries the request once with a refold (fold + clamped output budget). Numbers stated in the error text are deliberately NOT extracted as a window source — error-text learning is unstable under relay rewrites/dialect drift and would keep masking the real defect (nothing knew the window). sglang/#570-era trusted forms are unchanged. - clamp: clampOutgoingOutput refolds against the resolved window; prepare* receive the refold override as their nativeWindow (their only consumer is the clamp), so a stated-window refold clamps against the stated window. With the window configured correctly the first forward's clamp already prevents the overflow (zero 400s). - guidance instead of learning: the #1569 warn names the knobs (dsh settings.yaml models[].contextWindow, config.json providers.<url>.models.<model>.context); a new one-shot warn fires when an overflow arms while the window came from the unconfigured fallback. - failure contract unchanged: an unrescuable request still fails with the ORIGINAL 400 verbatim — once, no loop. Tests: #1812 e2e pins the three-way contract (configured → clamp prevents; sglang-stated → fold+clamp rescue; vLLM unconfigured → one forward, honest 400, nothing learned). context-overflow vLLM cases updated. Originally authored as e27e14b by ework-agent; reworked per the review decision to drop error-text extraction.
📦 Built Package ArtifactBranch: Option A — Install this exact build (recommended)npm install -g billion-context@0.0.1824-pr.1838.d73120aThis exact version is immutable and reproducible. The rolling Option B — Download artifact
tar xzf billion-context-pr1838.tgz
npm install -g packageInstall instructions are refreshed on each push. |
[bot] 🏷 ## PR #1838 Review 结论 方向关(审原 issue): #1812 失败链 = relay-fronted vLLM 真实 131k 窗口 × bili 未配置的 200k fallback → 压缩没及时触发 → vLLM 措辞 层次判断: 表面症状是「400 硬循环」,根因分层为 (a) 检测器方言覆盖缺口(本 PR);(b) sizing 漂移(fallback 200k vs 真实 131k,#1836 + 运维配置,不在本 PR)。PR 刻意不把错误文本数字当 window 源(relay 改写/方言漂移下不稳定),改为一次性指向配置旋钮的 warn——打的是根因不是症状。我逐条核过 代码审查要点:
本地验证(定向;全量回归在 CI):
测试有效性:断言有牙——A2 钉死两次 forward 的精确 wire body(第一次 max_tokens=65536 未动、第二次 clamp 后可证 ≤ stated window)+ arm 状态 + diff 整洁度:5 文件 +322/-21,每行都服务于本目标,无 lockfile/version/顺带改动。✅ 结构化七问
合并顺序提示: #1818(fold integrity,open)同样动 判定:可合(human merge)。方向正确、根因到位、回归复现证据完整(A2/B master 红 → 分支绿)、CI 全绿、契约保持。 |
|
(ai: zhipuai-lb/glm-5.3) 孤儿票清理:标题指向的 issue 已关闭,修复已由其他路径落地。关单;误判请 reopen。 |
Fixes the proxy side of #1812. The dsh-side handshake latch is fixed separately in #1836; the remaining fold-integrity defects (ex-#1819/#1820/#1775/#1800) stay in #1818 for review.
The failure chain (#1812)
Relay-fronted vLLM model, real window 131072, sized against bili's unconfigured 200k fallback (handshake gap → #1836). Compression never fired in time; when the server rejected with vLLM's phrasing —
— the overflow detector did not match it (no "window"/"context length" wording), so bili passed the 400 through verbatim and the client retried identical requests into a hard loop.
What this PR does
clampOutgoingOutputrefolds against the resolved window;prepare*receive the refold override as theirnativeWindow(its only consumer is the clamp). With the window configured correctly, the first forward's clamp already prevents the overflow — zero 400s.dsh settings.yaml models[].contextWindow,config.json providers.<url>.models.<model>.context); new one-shot warn when an overflow arms against the unconfigured fallback.Tests (e2e, hermetic fake upstream)
max_tokens, 200, zero 400s;Originally authored as
e27e14b8by ework-agent; reworked per the review decision to drop error-text extraction.