Skip to content

[PR] fix: parse LiteLLM input and completion overflow counts - #5969

Merged
ranxianglei merged 3 commits into
ranxianglei:masterfrom
zouzh22:2026-11-08_litellm-overflow-counts
Nov 9, 2026
Merged

ranxianglei merged 3 commits into
ranxianglei:masterfrom
zouzh22:2026-11-08_litellm-overflow-counts

Conversation

@zouzh22

@zouzh22 zouzh22 commented Nov 8, 2026

Copy link
Copy Markdown
Contributor

Fixes #5967

LiteLLM/SGLang context-overflow errors use requested a total of N tokens: M tokens from the input messages and C tokens for the completion. The existing parser recognizes the window but loses all three counts, so overflow recovery cannot use the stated input measurement or completion reservation.

Add this narrowly anchored variant alongside the existing parenthesized OpenAI/DeepSeek format. Reuse the existing calibration and headroom mechanisms; retain incomplete/error-status negative controls. Both plugin and proxy modes share the parser. Add bilingual technical notes.

Validation

  • Confirmed the parser gap at v0.1.212 and the checked upstream base 82e2d53.
  • Negative control before the fix: new unit regression fails on missing requestedTotal; new fake-upstream regression fails on missing completion reservation.
  • Rebased independently onto upstream 82e2d53; all 104 related tests below pass again on this base.
  • 65 related unit/calibration tests pass.
  • 7 fake-upstream calibration/refold tests pass.
  • 32 wire-contract/preflight/headroom tests pass.
  • npm run typecheck passes.
  • npm run build passes (dist import annotations and external-dependency checks included).
  • git diff --check passes.
  • Tests use the repository's hermetic preload and random loopback ports, never the user's production state.
  • The real-client/real-model Codex suite was not run: no test upstream is available at the default loopback endpoint; its ordinary invocation was skipped by the existing gate. CI is still pending.

Scope

This fixes missing error-format parsing, not DSH fork-state reconstruction. DSH intentionally omits parentSession on ordinary forks; compatibility and block-adoption behavior require a separate investigation. No claim that this parser patch alone restores every affected fork. No change to kernel, config, persistence, declared model windows or outgoing message contents.

All new fixtures use synthetic values. Human review/merge only.

@ranxianglei ranxianglei changed the title fix: parse LiteLLM input and completion overflow counts [PR] fix: parse LiteLLM input and completion overflow counts Nov 8, 2026
@ranxianglei ranxianglei added bug Something isn't working p1 labels Nov 8, 2026
…leanup)

Line 111 asserted st.overflowArmCompletionReserve === 5000, identical to the
pre-existing line 136 (ranxianglei#3450). Removing the redundant copy; coverage is
unchanged.
@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Note: I initially posted these findings to the wrong thread number (#5969) due to a mirror-vs-GitHub numbering mix-up. This #5971 is the correct thread — treat this as the authoritative review and disregard the stray copy.

Reviewed against live master (merge-base 82e2d53, matching your declared base). The fix is correct, correctly scoped, CI is green, and it's safe to merge. I made one trivial cleanup commit on your branch.

Dup screening

Not a duplicate. Searched open + closed issues/PRs for the LiteLLM colon wording and parseOverflowCounts:

What I verified

  • Parser correctness. Extracted the exact HEAD functions and ran them against 8 fixture groups → 50 passed, 0 failed: LiteLLM plain + comma-grouped (total/messages/completion all populated), HTTP-429 → not overflow, the 3 incomplete-split negatives (missing completion / missing "tokens from" / "output tokens" instead of "for the completion") → counts stay undefined while the window still parses, plus OpenAI-parens and Anthropic regressions still passing.
  • "Both modes share the parser." True: inspectContextOverflow is a pure util consumed at src/server/relay.ts:1212/1444/1634; armOverflowShrink (src/server/relay.ts:1260) maps requestedMessages→calibration baseline and requestedCompletion→overflowArmCompletionReserve. relay.ts is the shared upstream-response pipeline for plugin and proxy modes.
  • Rebase safety. git diff --numstat <merge-base>..origin/master over all 5 touched files = 0 lines changed on master → zero semantic-conflict risk despite behind.

Cleanup I pushed (one line, on your branch)

tests/e2e-issue3900-overflow-teaches.test.ts line 111 asserted st.overflowArmCompletionReserve === 5000, duplicating pre-existing line 136 (…=== 5_000, "#3450"). Removed the redundant copy — coverage unchanged, authorship preserved (fast-forward, no force): 8b25694 → 911737c.

Regression / existing-behavior check

New branch sits after the OpenAI-parens pattern and before Anthropic/Gemini/CJK → can't shadow any earlier match; tight anchor means no over-match (throttle-400 and the incomplete splits don't match). No behavior change beyond adding the missing parse. No kernel/config/persistence/wire-shape/outgoing-content change.

CI (live, head 911737c)

All checks complete: 36 passed, 12 skipped, 0 failed. The 12 skips are the gated real-client e2e lanes (no upstream at loopback), expected. Full test matrix green.

Labels

bug + p1 present and appropriate. Judged on the diff I'd also tag dsh (repro host) and scope/wiring (upstream-response parsing boundary); my API attempts to add those two didn't persist in this environment (verified live), so they may need a manual tag. On priority: by the triage scale this reads closer to p2 (degraded recovery precision with a working window-based fallback — not availability/data-loss/cache-leak); currently p1 as set — flagging for your call rather than silently downgrading.

Verdict

Mergeable — CI green on 911737c. Merge stays human-only: #5969


八问(结构化审核结论)

  1. 是什么问题:LiteLLM/SGLang 的上下文溢出错误会给出三个计数(请求总量、输入消息占用、预留补全量),现有解析器只识别到窗口、把这三个数全丢了,导致溢出恢复拿不到上游声明的输入实测值和补全预留。
  2. 什么原因导致的:parseOverflowCounts 只覆盖了 OpenAI/DeepSeek 括号形和 Anthropic/Gemini/CJK 分支;LiteLLM 冒号形没有对应分支,于是 total/messages/completion 保持 undefined(窗口由独立的 parseOverflowWindow 仍能测出)。
  3. 影响范围与严重程度:影响经 LiteLLM/SGLang 后端溢出的会话(复现打的是 [dsh])。严重度:恢复精度降级——校准 k̂ 样本与输出-headroom 的补全预留无法从声明计数生效,仍有基于窗口的兜底 → 按分级口径偏 p2(当前标 p1,已提请 owner 确认);不属于可用性/数据丢失/缓存泄露类。
  4. 怎么修复的:新增一条紧锚定正则分支,置于 OpenAI 括号模式之后、Anthropic/Gemini/CJK 之前,复用既有 toTokenNumber 与返回结构;不引入新机制。
  5. 是否有回归问题:顺序上不会遮蔽更早的模式;锚定足够紧不会误匹配(throttle-400 与三条不完整切分负例均不命中);OpenAI 括号形与 Anthropic 回归仍通过;全部 5 个改动文件在 merge-base..master 区间为 0 行改动 → rebase 干净。除新增缺失解析外无行为变化。
  6. 是否破坏缓存:否。纯函数、无状态,未触碰任何缓存/重试/游标/去重键。
  7. 是否改了设计方向:否。复用既有校准([Bug][dsh native] preflight fold target sits ABOVE the engine's own denominator (1048576 vs 1000000) and the upstream-stated window from a 400 never feeds it — 'deficit 0' forwards an over-window payload → context_length_exceeded kills the turn #3450)与 headroom 机制,无架构/边界/数据流转向。
  8. 是否引入新配置:否。无 env/config/CLI 变更。

@ranxianglei
ranxianglei merged commit 1bc2faf into ranxianglei:master Nov 9, 2026
27 checks passed
@github-actions

github-actions Bot commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

📦 Merged Build Published

This merge to master was published as 0.0.1109-master.1bc2faf (rolling master tag).

npm install -g billion-context@0.0.1109-master.1bc2faf  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

Louie0806 added a commit to Louie0806/billion-context that referenced this pull request Nov 9, 2026
…ndow refusal, ranxianglei#5969 LiteLLM overflow counts

Advances the base from 1f311dd to 39fc2ee (8 commits, 8 files). The only
file this lane shares with the delta is scripts/live-files.mjs, and both sides
are byte-identical (md5 295a1317030b28dbff36539f8e97d176): upstream landed the
same ranxianglei#1248 registration this lane already carried from the ci-check commit
156d0c8, so the merge is a no-op there and git auto-resolved everything else.

Notable upstream commits adopted:
 - 4ec9778 fix: empty-slot direct send now refuses over-window bodies locally (ranxianglei#5996)
 - 12b18e9 fix(native-intercept): name the floored local bar distinctly from the
   resolved window in the empty-slot refusal (ranxianglei#5996 review)
 - 8b25694 fix: parse LiteLLM input and completion overflow counts (ranxianglei#5969)
 - 911737c test: drop duplicate overflowArmCompletionReserve assertion
 - 831b645 fix(ci): register tests/issue5955-wedged-same-lane-takeover.test.ts in LIVE_FILES
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working p1

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug][dsh] LiteLLM overflow counts do not reach calibration or output-headroom recovery

2 participants