Skip to content

[PR] fix(#1812): recognize vLLM overflow errors — honest rescue, configuration is the sizing source - #1838

Closed
ranxianglei wants to merge 3 commits into
masterfrom
2026-10-01_vllm-overflow-recognize
Closed

ranxianglei wants to merge 3 commits into
masterfrom
2026-10-01_vllm-overflow-recognize

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

Fixes the proxy side of #1812. The dsh-side handshake latch is fixed separately in #1836; the remaining fold-integrity defects (ex-#1819/#1820/#1775/#1800) stay in #1818 for review.

The failure chain (#1812)

Relay-fronted vLLM model, real window 131072, sized against bili's unconfigured 200k fallback (handshake gap → #1836). Compression never fired in time; when the server rejected with vLLM's phrasing —

prompt (125000 tokens) + max tokens (65536) exceeds the context (131072); requests are never truncated

— the overflow detector did not match it (no "window"/"context length" wording), so bili passed the 400 through verbatim and the client retried identical requests into a hard loop.

What this PR does

Tests (e2e, hermetic fake upstream)

  1. configured real window + oversized turn → first forward clamps max_tokens, 200, zero 400s;
  2. sglang-stated window + declared 200k → 400 → fold/clamp against the stated window → one retry → 200;
  3. vLLM dialect unconfigured → one forward, honest verbatim 400, armed at the payload estimate, no window learned, no loop.

Originally authored as e27e14b8 by ework-agent; reworked per the review decision to drop error-text extraction.

ework-agent and others added 2 commits October 1, 2026 15:43
…t to the stated window

vLLM rejects oversized requests with 'prompt (N tokens) + max tokens (M)
exceeds the context (W); requests are never truncated' — a phrasing without
the word 'window', so inspectContextOverflow never matched it. The 400 passed
through verbatim and clients that treat it as fatal locked the session in a
400 retry loop even when the payload was rescuable (#1812 log: declared
window 200000 vs real 131072).

Three coordinated changes:
- util: detect the vLLM dialect, extract the stated window W and the stated
  prompt size N (new ContextOverflowInfo.promptTokens).
- server/budget: the overflow-refold rebuild's output clamp now runs against
  min(declared, stated window) with a FRESH input estimate floored by the
  upstream's own prompt count. Previously it ran against the declared window
  while lastInputTokens was armed to the stated window (#987), which either
  left the client's max_tokens untouched (input alone fit) or zeroed the cap.
  The normal (non-refold) clamp path is byte-for-byte unchanged.
- server: thread the stated window + prompt count through the refold rebuild
  (runPreparedPipeline -> prepare{Openai,Google,Responses} -> clamp).

Tests: dialect unit cases (verbatim log message, comma grouping, sglang
negative for promptTokens) and a hermetic e2e pinning the full chain on the
OpenAI wire: reject -> arm -> refold clamps max_tokens -> re-send once in
the same request -> 200, provably fitting the real window.
…tion is the sizing source

The #1812 incident, decomposed: a relay-fronted vLLM model with a real
131072 window was sized against bili's unconfigured 200k fallback
(handshake gap — dsh side fixed in #1836), so compression never fired in
time; when the server finally rejected with vLLM's phrasing — 'prompt
(N tokens) + max tokens (M) exceeds the context (W); requests are never
truncated' — the detector did not match it (no 'window'/'context
length' wording), so bili passed the 400 through verbatim and the client
retried identical requests into a hard loop.

This PR fixes the proxy side:

- recognition: the vLLM dialect matches CONTEXT_OVERFLOW_PATTERN →
  arms the #1110 shrink and retries the request once with a refold
  (fold + clamped output budget). Numbers stated in the error text are
  deliberately NOT extracted as a window source — error-text learning
  is unstable under relay rewrites/dialect drift and would keep masking
  the real defect (nothing knew the window). sglang/#570-era trusted
  forms are unchanged.
- clamp: clampOutgoingOutput refolds against the resolved window;
  prepare* receive the refold override as their nativeWindow (their
  only consumer is the clamp), so a stated-window refold clamps
  against the stated window. With the window configured correctly the
  first forward's clamp already prevents the overflow (zero 400s).
- guidance instead of learning: the #1569 warn names the knobs (dsh
  settings.yaml models[].contextWindow, config.json
  providers.<url>.models.<model>.context); a new one-shot warn fires
  when an overflow arms while the window came from the unconfigured
  fallback.
- failure contract unchanged: an unrescuable request still fails with
  the ORIGINAL 400 verbatim — once, no loop.

Tests: #1812 e2e pins the three-way contract (configured → clamp
prevents; sglang-stated → fold+clamp rescue; vLLM unconfigured → one
forward, honest 400, nothing learned). context-overflow vLLM cases
updated. Originally authored as e27e14b by ework-agent; reworked per
the review decision to drop error-text extraction.
@github-actions

github-actions Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-01_vllm-overflow-recognize (5054cfd)

Option A — Install this exact build (recommended)

npm install -g billion-context@0.0.1824-pr.1838.d73120a

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr1838.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 收到,开始按 PR review 规范审 #1838:先核对原 issue #1812 方向、拉分支对代码、看 CI 状态。结论会按结构化七问回帖。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 ## PR #1838 Review 结论

方向关(审原 issue): #1812 失败链 = relay-fronted vLLM 真实 131k 窗口 × bili 未配置的 200k fallback → 压缩没及时触发 → vLLM 措辞 prompt (N tokens) + max tokens (M) exceeds the context (W) 无 "window" 字样、旧 pattern 不匹配 → 400 原样透传 → 客户端原样重试成硬循环。本 PR 修 proxy 侧识别 + 救援;dsh 握手缺口由 #1836 单独修(已 closed),fold-integrity 缺陷留在 #1818(open)——范围切分与仓库现状核对一致,无重叠、无越界。✅

层次判断: 表面症状是「400 硬循环」,根因分层为 (a) 检测器方言覆盖缺口(本 PR);(b) sizing 漂移(fallback 200k vs 真实 131k,#1836 + 运维配置,不在本 PR)。PR 刻意不把错误文本数字当 window 源(relay 改写/方言漂移下不稳定),改为一次性指向配置旋钮的 warn——打的是根因不是症状。我逐条核过 parseOverflowWindow 全部既有 regex 对 vLLM 措辞均不命中 → window=undefined,「不学习」决策在代码层面成立而非仅声明。

代码审查要点:

  • master 上 refold 只对 preflightCompressIfNeeded 覆写 reqConfig.modelContextLimit,prepare* 的输出 clamp 仍拿配置窗口 → 即使输入折到 stated 之下,max_tokens 原样重发,retry 照样溢出。本 PR 让 prepare* 收 overflowWindow ?? nativeWindow,且 refold 时 fresh 测输入(estimateFreshInputTokens,忽略 baseline——arm 已把 lastInputTokens 写成窗口值,沿用会把输出预算挤到 ~0)。两处改动互补且必须同时成立。
  • 失败契约不变:不可救 → 原始 400 原样、仅一次、无循环(单次直接 fetchWithTimeout 重试,非递归;rebuild 后 body 逐字节相同则返回 null → 直接透传)。
  • prepareAnthropic 本就不带 window 参数、不走此 clamp 路径,未改——正确。
  • estimateInputTokens 非 refold 路径重构后语义字节等价(max(baseline, est) 不变)。

本地验证(定向;全量回归在 CI):

  • typecheck ✅;tests/context-overflow.test.ts + tests/issue1812-vllm-overflow.test.ts 共 30/30 ✅(含全部既有 case)。
  • 回归复现证明:新 e2e 在 master 上跑——A 通过(正确配置守卫,master 即有)、A2 失败、B 失败;在分支上 3/3 通过。A2 正是核心回归 case(master:refold clamp 用 200k → max_tokens 不动 → retry 被拒 → 客户端见 400)。
  • CI:13 项 check 全绿(ubuntu/windows ×2 全量单测、real pi/opencode/codex/dsh lanes、e2e、hermetic registry、version-guard)。

测试有效性:断言有牙——A2 钉死两次 forward 的精确 wire body(第一次 max_tokens=65536 未动、第二次 clamp 后可证 ≤ stated window)+ arm 状态 + confirmedContextLimits === undefined(不学习);B 钉死 400 原样 + 恰好一次 forward + lastInputTokens !== REAL_WINDOW(将来谁再引入 error-text 提取会立刻被抓)。全确定性(listen(0) loopback + hermetic stubs),无假测试。

diff 整洁度:5 文件 +322/-21,每行都服务于本目标,无 lockfile/version/顺带改动。✅

结构化七问

  1. 是什么问题:vLLM 系 "exceeds the context (W)" 400 不被 overflow 检测器识别 → 原样透传 → 客户端硬循环;且即便识别,refold 的输出 clamp 仍按错误窗口计量,重试必然再失败。
  2. 什么原因导致的:两层——(a) 检测 pattern 方言覆盖缺口;(b) refold 路径输出 clamp 用配置窗口 + stale baseline,与 upstream stated 窗口不一致。上游 sizing 漂移属 fix(dsh): retry failed model-window resolves — one boot race no longer strips x-bili-plugin-context-window #1836/运维配置,分层处理正确。
  3. 怎么修复的:补识别 pattern(arm 救援);refold 把 stated 窗口贯穿到 prepare* clamp 并 fresh 测输入;vLLM 数字刻意不学习,以一次性配置旋钮 guidance warn(source=default)替代。
  4. 是否有回归问题:审过的既有行为——各既有 dialect 识别/window 提取(sglang/llama.cpp 等,既有单测钉住,全过)、非 refold clamp 语义(字节等价)、armOverflowShrink 主体逻辑(仅加 warn)、失败契约(原样一次、无循环,代码路径核实)、[OpenCode v2] Configured model context limit never reaches the proxy (x-bili-plugin-context-window not stamped) - silent registry-peek fallback suppresses proactive nudge and misfires emergency shrink #1569 warn 触发条件(不变,仅文案扩)。无意回归:无。有意行为变更披露:vLLM 方言 400 现在进入救援路径(旧:纯透传);refold retry 的 max_tokens 可能被 clamp(旧:原样重发)——两者即修复本身。
  5. 是否破坏缓存:否。prefix-affinity/增量游标/去重键均未触碰;overflowGuessWarned 为进程级内存 Set,一次性去重、不落盘。
  6. 是否改了设计方向:否。仍在 context window problem continues #987/「BUG」sideRequestGuard 会永久拦截进程内第三方插件的 LLM 调用,导致与MemOS冲突 #1110/使用pi 压缩后就提示prompt is too long #1195 救援架构内;明确拒绝 error-text 学习,坚持「配置是唯一 sizing source」。
  7. 是否引入新配置:否。无新 env/config field/默认值;warn 文案只指认既有旋钮(providers.<url>.models.<model>.context、dsh settings.yaml models[].contextWindow)。

合并顺序提示: #1818(fold integrity,open)同样动 src/server.ts——两者先合者需另一个 rebase;建议先合本 PR(范围小、CI 全绿),#1818 rebase 后再审。另:GitHub 此刻 mergeable 状态在重算中(此前为 True),合并前请确认 merge 按钮可用。

判定:可合(human merge)。方向正确、根因到位、回归复现证据完整(A2/B master 红 → 分支绿)、CI 全绿、契约保持。

@ranxianglei

Copy link
Copy Markdown
Owner Author

(ai: zhipuai-lb/glm-5.3)

孤儿票清理:标题指向的 issue 已关闭,修复已由其他路径落地。关单;误判请 reopen。

@ranxianglei ranxianglei changed the title fix(#1812): recognize vLLM overflow errors — honest rescue, configuration is the sizing source [PR] fix(#1812): recognize vLLM overflow errors — honest rescue, configuration is the sizing source Oct 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant