Skip to content

fix: forward over-window payload when preflight is exhausted (#333) - #334

Closed
ranxianglei wants to merge 1 commit into
masterfrom
2026-08-29_preflight-exhausted-forward
Closed

ranxianglei wants to merge 1 commit into
masterfrom
2026-08-29_preflight-exhausted-forward

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

What

Fixes the preflight fail-fast × soft-protection livelock (#333): when the payload is over-window and there is nothing left to fold (all remaining messages sit in the soft-protected recent/last-user zone), the proxy used to fail-fast 502 on every retry — a permanent loop, because that state is static and the EMERGENCY nudge can never reach the model (fail-fast is exactly what blocks the request from arriving).

Implements fix direction A from the issue (author's stated preference: minimal semantic break):

  • src/server.ts preflightCompressIfNeeded — both exhausted sites now forward the payload as-is with a warning instead of fail-fast 502:
    1. no compressible ranges before preflight starts (the short-session case from the issue: m00004/m00005 all protected);
    2. preflight ran and returned failure.kind === "exhausted" (budget exhausted / ranges no longer resolve / nothing viable). If preflight partially compressed first, the rebuilt payload (with the partial compression kept) is forwarded.
  • Fail-fast is kept for transient states: summary-upstream failures (429 → 503 + Retry-After: 30, other → 502) and client abort. Those self-recover on retry; exhaustion does not.

Why forwarding is safe (and heals)

The window limit bili enforces is often a fallback/learned/relay estimate, not ground truth. Forwarding lets the truth decide:

  • Upstream accepts (real window ≥ payload): the session is alive, and the injected EMERGENCY nudge lets the model self-heal via compress — the recovery path fail-fast had cut off.
  • Upstream rejects (real window < payload): the client sees the upstream's truthful 400 (passed through verbatim by the existing error path) instead of an opaque bili 502, so it can converge (surface the error / new session). The existing overflow self-heal also learns the real window from the 400 and arms an emergency shrink for the next turn — machinery that fail-fast had made unreachable in this state.

Tests

tests/preflight-fail-fast.test.ts:

  • rewritten the old "nothing compressible → 502, no forward" test → now asserts the over-window payload is forwarded (200 from an accepting upstream, no summary call spent);
  • new e2e: same static state but the upstream's real window is smaller → asserts the upstream 400 is passed through verbatim to the client and the self-heal learns the real window (learnedContextLimits["claude-small"] === 8000);
  • the two fix: honor anthropic-beta context-1m to size the context window #301 transient-failure tests (429 → 503; fitting payload still forwarded) are unchanged and still pass.

Pre-flight

  • npm run typecheck — clean
  • npm test — 746/746 pass
  • npm run build — clean

@github-actions

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-08-29_preflight-exhausted-forward (f1076e5)

Option A — Install from npm PR tag (recommended)

npm install -g billion-context@pr-334

Each push to this PR publishes a new version under the pr-334 npm tag.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr334.tgz
npm install -g package

This comment is automatically updated on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 ework agent · vllm-qwen/qwen3.8-27b

[bot] Reviewing PR #334 now — fetched head f1076e5 (based on current master 3c3778f). Diff is small: src/server.ts (+31/−?) and tests/preflight-fail-fast.test.ts. I'll read the full diff, verify the two exhausted sites + kept transient fail-fasts, then run typecheck/test/build locally and report back.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 ework agent · vllm-qwen/qwen3.8-27b

[bot] Review 完成,LGTM ✅

在 PR head f1076e5(parent = master tip 3c3778f,工作区干净)上独立复核:

代码与描述一致

  • 两处 exhausted 站点均改为带 warning 转发:
    • src/server.ts:1802-1815 — preflight 启动前就无可折范围(payloadEstimate >= limit 时)→ return prepared
    • src/server.ts:1859-1875 — failure.kind === "exhausted" → 有 compressedRanges > 0 时 runPrepare() 重建转发(与既有 success 路径 src/server.ts:1844-1847 完全同构,含 stats.requests -= 1 修正);否则原样转发 prepared
  • 瞬态 fail-fast 保留:aborted → 不响应(src/server.ts:1855-1857);upstream 429 → 503 + retry-after: 30;其他 upstream → 502(src/server.ts:1878-1879)

self-heal 链路真实可达(新测试断言有据)

  • 400 透传走既有 !upstream.ok 分支(src/server.ts:2049-2116)→ inspectContextOverflow(src/util.ts:152);CONTEXT_OVERFLOW_PATTERNS 含 context_length_exceeded / maximum context length,parseOverflowWindow(src/util.ts:134-147)匹配 maximum context length is (\d+) → 8000 → 写入 learnedContextLimits[reqModel] 并 arm emergency shrink。测试断言的 learnedContextLimits["claude-small"] === 8000 正是这条真实路径的产物。

无回归

  • 全库检索确认没有其他测试依赖旧的 exhausted→502 行为(tests/fix-config-session.test.ts 里的 "exhausted" 是 session pool,无关)。

Pre-flight(独立重跑)

一个已知权衡(非阻塞,仅记录):若 bili 的窗口限制恰好是 ground truth(非 fallback/learned 估计),转发会每次重试多消耗一次注定 400 的上游请求。但系统会收敛(客户端拿到真实 400 + self-heal 学到真实窗口),而非旧行为的永久 livelock——与 #333 里你选定的方向 A(最小语义破坏)一致,可接受。

可以合并(按规则由你操作)。

@ranxianglei ranxianglei closed this Sep 5, 2026
@ranxianglei

Copy link
Copy Markdown
Owner Author

Closing as superseded by PR#337, which implements the issue's preferred fix direction (relax the soft-protected recent zone oldest-first, hard protectedTools still honored).

This PR took the issue's explicitly-"weaker alternative" (forward-as-is and let the upstream arbitrate). It also references #333, which is a different issue (compaction_trigger mutation) already fixed by the native-compact interception.

Nothing against the implementation itself — reviewed clean at the time — it's just the wrong direction to keep alongside #337, and the two are semantically exclusive.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant