Skip to content

Preflight compression holds the request with zero response bytes; client undici headersTimeout (default 300s) kills it -> repeated [preflight] summarization aborted: client disconnected 5-minute death loop, payload never forwarded #568

Closed

Description

@shengbuding

Environment

  • billion-context v0.1.86 (npm), proxy mode (bili start --port <local>), Windows.
  • Client: dsh (DeepSeek Harness) CLI talking HTTP to the proxy's /bili/<upstream> origin; upstream = OpenAI-compatible remote endpoint (deepseek-v4-flash), also reproduced feeding a local llama.cpp server.
  • Setups both drive the session well over the model window (e.g. ~954K tokens vs effective window 653,896) so every request enters preflight.

Symptom

When the session context exceeds the model window, the proxy runs preflight compression before emitting any response bytes to the client. Compression of 700K鈥?50K-token histories takes 240鈥?07 seconds (summarization round-trips against the upstream model). The client's HTTP stack (Node undici, default headersTimeout = 300,000 ms) aborts the request while preflight is still running; the proxy then records:

[preflight] summarization aborted: client disconnected
[<session>] preflight aborted (the client disconnected during preflight compression); not forwarding

and never forwards. The client retries; the retry re-enters preflight; same 300s wall. Observed abort cadence is ~5 minutes apart, exactly matching undici's default 300 s headersTimeout, for 6 consecutive rounds (one session, 2026-09-06 UTC):

02:46:51  [preflight] summarization aborted: client disconnected 鈥?not forwarding
02:52:24  (same)         (+5m33s)
02:57:32  (same)         (+5m08s)
03:02:39  (same)         (+5m07s)
03:14:14  (same)         (+11m35s, compression round was longer)
03:19:22  (same)         (+5m08s)

Same failure repeated across three different sessions and both upstream types (remote + local llama.cpp), 10+ occurrences in one day. A successful (client-side, in-session) compression that shrunk the history 88% was the only thing that ended the loop; after that the session ran normally.

Adjacent evidence from the same window: a preflight that did finish shows the timescale 鈥?preflight compressed 2 range(s), ~7435 tokens saved (702031 鈫?694596) in 265882ms; rebuilding payload 鈥?265,882 ms is within spitting distance of the 300 s client timeout, and that very round is one of the aborted ones (the client died at ~300 s while compression was still running).

Root cause (as observed, no suggested fix)

  • Server side, on entering preflight compression the proxy sends no response headers to the client until compression completes and the rebuilt payload is ready.
  • Client side, Node's undici default headersTimeout is 300,000 ms (no headers received within 300 s => connection aborted with UND_ERR_HEADERS_TIMEOUT).
  • Preflight compression of very large histories routinely takes 240鈥?07 s 鈥?overlapping the client's timeout window, so the round fails by racing the client's patience rather than by any compression result.
  • Mid-flight detection exists (log line summarization aborted: client disconnected), so the proxy knows the client is gone but only after spending the whole compression budget.

Evidence / reproduction

  • bili.log (UTC): lines above + context 954192 tokens exceeds model window 653896 (model=deepseek-v4-flash); preflight compressing before forward before each round.
  • Preflight durations observed (same proxy version): 265,882 ms / 124,083 ms / 149,045 ms / 107,415 ms / 84,267 ms / 56,422 ms / 48,006 ms 鈥?several 鈮?the client's 300 s cap.
  • Also seen with the local upstream: context 164498 tokens exceeds model window 126645 鈥?preflight compressed 1 range(s), ~42683 tokens saved (164498 鈫?121815) in 124083ms 鈥?124 s compression, then the forwarded request was truncated upstream (separate issue), after which the client-side retry again entered preflight and got aborted at the 300 s wall again.

Outcome if unaddressed

Any client with a default 300 s header timeout that exceeds the model window in a big session is stuck in a guaranteed, repeating preflight-abort loop (5-minute cadence); the proxy neither forwards nor returns any error to the client 鈥?it just aborts internally, making the session appear to "hang every 5 minutes".

(Log excerpts redacted to remove session identifiers, ports and keys.)

Activity

  1. ranxianglei commented on Sep 6, 2026

    @ranxianglei
    Owner

    🤖 ework agent · qwen3.8-27b

    [bot] 🏷 Triaged — root cause confirmed in code, fix in progress.

    Verification (per triage rules):

    • Duplicate screen: searched all open+closed issues for preflight/timeout/headersTimeout/aborted — no prior report of this.
    • Root cause is exactly as you diagnosed, confirmed in src/server.ts: preflightCompressIfNeeded() runs inside the per-session lock, before forward(), and the first response byte (even the status line) is only written when the upstream answers (res.writeHead in forward(), src/server.ts:2897/2903). During preflight the client sees a silent socket. undici's default headersTimeout (300s) therefore races your 240–407s compress rounds and wins whenever a round-trip chain crosses it — hence the exact 5-minute cadence and the summarization aborted: client disconnected pair in the logs. Your "one finished preflight took 265,882ms" line is the smoking gun: it's inside the 300s window by ~35s of luck.
    • Layering: symptom = abort/retry loop; direct cause = zero bytes emitted before the client's header deadline; root cause = preflight blocks the response lifecycle with no early commitment. Mid-flight detection (ABORTED_FAILURE) is working as designed — it just fires after the client already gave up.

    Fix design (branch in progress):
    When preflight compression is still running past a grace period (default 30s, BILI_PREFLIGHT_HOLD_MS), the proxy commits the response early:

    • streaming requests: 200 + text/event-stream immediately, then SSE keep-alive comment lines (: bili-preflight) every 15s while compression runs (same pattern OpenAI uses with its SSE pings — comments are spec-mandated-noop for every SSE consumer),
    • non-streaming requests: 200 + application/json with periodic whitespace chunks (valid JSON padding).

    Once bytes flow, undici switches from headersTimeout to bodyTimeout (inactivity-based, reset by each keep-alive chunk), so the client is held for the full compression duration regardless of how many rounds it takes. The moment preflight finishes, the normal response pipeline runs unchanged.

    Deliberate tradeoff, documented in the PR: only requests whose preflight exceeds the grace period lose HTTP status-code fidelity on failure (the #301 fail-fast 502/503 becomes a 200 carrying the same structured error in-band — native error event for SSE, same JSON error body for non-stream). Fast paths (including quick fail-fasts) are byte-identical to today, so the existing #301 contract and its tests are untouched.

    Will post the PR link here once tests + build pass.

  2. ranxianglei commented on Sep 6, 2026

    @ranxianglei
    Owner

    🤖 ework agent · qwen3.8-27b

    [bot] 🏷 Fixed — PR #571: #571

    Verification (per triage): confirmed the root cause in code, not just from the logs. handle() in src/server.ts runs preflight before any response byte exists; the only mid-flight detection is the res "close" abort ([preflight] summarization aborted: client disconnected), which fires only after the client's full header budget is spent — exactly your 240–407s durations vs undici's 300s default. Your one finished run at 265,882ms shows how close the boundary is. The analysis in the issue is the root cause itself (zero-byte silence), not a symptom — no deeper layer to chase.

    The fix — "preflight hold": when preflight outlives a grace period (default 30s, tunable via BILI_PREFLIGHT_HOLD_MS), the proxy commits the response early and holds you with keep-alive bytes:

    • streams: 200 + text/event-stream + SSE comment keep-alives (: bili-preflight, every 15s — a spec-mandated no-op for every SSE consumer, same pattern as OpenAI's pings)
    • non-stream: 200 + application/json + whitespace padding (valid JSON)
    • early commits carry an x-bili-preflight: compressing header so you can see it happening in logs/captures

    Each byte resets undici's inactivity-based bodyTimeout, so the client survives compressions of any duration. Both plugin mode (ACP clients pipe verbatim: comments/whitespace are inert) and proxy mode work.

    Documented trade-off: once the status is committed, a late failure can't change it anymore — so it's delivered in-band instead: a protocol-native error event (preflight_compress_failed) for streams, or the identical structured error body under the already-sent 200 for JSON. Fast paths (< 30s) are completely unchanged and keep full status-code fidelity — all existing #301 fail-fast tests pass untouched.

    Pre-flight: typecheck clean · build clean · 1051/1052 tests pass (the 1 failure is a pre-existing environment-dependent launcher test that also fails on unmodified master here) · new tests/preflight-hold.test.ts adds 4 e2e cases with timing assertions proving headers arrive before the slow summarization finishes · E2E zero-token check passed; the full ACP_TEST_E2E codex run needs a reachable upstream that this environment can't reach, so please run it (or the CI dispatch) before merging.

    For your immediate setup: after this ships, the dsh CLI will see 200 within ~30s of each request and stream normally once compression finishes — no more 5-minute abort loop.

  3. ranxianglei commented on Sep 7, 2026

    @ranxianglei
    Owner

    修复已随 PR#571 合并 (eb0f532): 预压缩超过宽限期(默认 30s, BILI_PREFLIGHT_HOLD_MS 可调)时代理提前提交 200 + 保活字节挂住客户端(SSE 注释 / JSON 空白), 压缩完成后同连接续传真实响应; 迟到失败带内投递。#568 死循环断根。手工关闭(标题缺 closing 关键字)。

  4. ranxianglei commented on Sep 7, 2026

    @ranxianglei
    Owner

    Fixed by #571 (merged in eb0f532, shipped semantics):

    • Preflight now commits the response early after a 30s grace (PREFLIGHT_HOLD_GRACE_DEFAULT_MS, overridable via BILI_PREFLIGHT_HOLD_MS): streaming clients get SSE 200 with : bili-preflight keep-alive comments, non-streaming clients get padded chunked JSON — every byte resets undici's inactivity-based bodyTimeout, so the 300s headersTimeout wall no longer kills the request mid-compression.
    • If preflight ultimately fails after the early commit (status line can no longer change), the failure is delivered in-band on the protocol's native error channel: top-level error + [DONE] (openai), event: error (anthropic/responses), or an identical JSON error body for non-stream — instead of a silent hang/abort loop.
    • Client disconnect during the hold still aborts preflight as before; quick preflights (< grace) are untouched (no early commit).

    Regression: tests/preflight-hold.test.ts (342 lines: grace, keep-alive cadence, in-band error per protocol, hold suppression on fast preflight).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions