Skip to content

preflight fail-fast × soft-protection livelock: over-window payload with nothing foldable outside the protected zone 502s forever #330

Closed

Description

@ranxianglei

Symptom

A session whose payload exceeds the model window and whose entire foldable content sits inside the soft-protected recent zone gets stuck in a permanent 502 loop. Every request fails fast before forwarding, and no recovery path exists — the request never reaches the model, so the EMERGENCY nudge (the only other recovery mechanism, #299) can never fire.

Found while building the e2e suite (#329): the suite's load-growth phase rides large user-messages on the wire verbatim, and under a forced small window (BILI_LAUNCHER_MODEL_WINDOWS) the recent zone ends up covering all foldable content.

Mechanism

  1. Preflight trigger: tokenCount >= config.modelContextLimit (src/server.ts:1788).
  2. The kernel's nudge computes compressibleRanges excluding the soft-protected recent zone (preserveRecentMessages, default 5, plus the last user message) — acp-kernel: "messages there are excluded from the range but do not fail it (an entirely-protected range fails with a clear error)".
  3. When nothing viable remains (>= 200 tokens per viableRanges):
    • src/server.ts:1802-1807 → failFast(502, "no part of the conversation is compressible (nothing left to fold)", false), or
    • src/preflight.ts:264-266 → failure = { kind: "exhausted", detail: "no compressible ranges remain in the conversation" } → src/server.ts:1853 → failFast(502, ..., false).
  4. The payload is never forwarded ("The over-window payload was NOT forwarded." — src/server.ts:1798). The model never sees it, so the EMERGENCY nudge / model-driven compress can never fire.
  5. Nothing changes between attempts (same session state, same payload, same window) → every retry is an identical 502. retryable: false tells the client it is a hard failure, but it is actually a permanent one — the session is bricked until the window is raised or history is trimmed externally.

This is the same stuck-session class that #247 fixed for model switches (preflight was that fix); #301's fail-fast introduced a new variant: when preflight itself cannot help, the proxy 502s instead of forwarding (where the upstream 400 would at least surface to the user and the reactive path could still try).

Impact

  • A session can be permanently stuck with no recovery path — worse than a transient error.
  • Likely in real operation: any workload where one large recent message (big tool output, big user payload) pushes context over a small forced window. The e2e suite hits this deterministically under a forced window.
  • Compounded by relay usage inflation (~2×, noted in Model: glm-5.3 — test: e2e suite — real codex through bili (warmup/load/compress/purity/forge) #329): the trigger fires earlier than true usage, making the state easier to reach.

Suggested fix

When the payload is over-window and no viable range exists outside the protected zone, fall back in order:

  1. Relax the soft protection (preferred): re-run preflight with the recent zone relaxed (oldest-first within the zone), still honoring the hard protectedTools exclusion. Soft protection is a preference, not a constraint — under overflow, folding the oldest soft-protected content beats bricking the session.
  2. Emergency truncation of large tool outputs inside the protected zone (the kernel already ships truncateLargeToolOutputs as a last-resort safety valve; it never touches summaries).
  3. If both are exhausted, keep the fail-fast 502 but make the error actionable: name the protected zone, the sizes involved, and what an operator can do (raise the window / restart the session).

Alternative (weaker): forward as-is when the payload's own estimate is close to the limit — the window estimate is approximate and relay usage inflation means true usage may fit — letting the reactive nudge handle it.

Repro (from the #329 e2e suite)

  1. Force a small window via BILI_LAUNCHER_MODEL_WINDOWS.
  2. Send a payload where a recent user-message alone exceeds the window (the suite's load-growth pattern: deterministic payload user-messages riding the wire verbatim).
  3. Observe on every request, indefinitely: preflight fail-fast 502 (retryable=false): context ~N tokens exceeds the model window W (model=M) and preflight compression could not bring it under: no part of the conversation is compressible (nothing left to fold). The over-window payload was NOT forwarded.

Reported by @ranxianglei while building #329 (report-only there; filed separately per repo convention).

Activity

  1. ranxianglei commented on Aug 29, 2026

    @ranxianglei
    OwnerAuthor

    🤖 ework agent · vllm-qwen/qwen3.8-27b

    [bot] Picked up. Reproducing the mechanism first: preflight's viableRanges(turn.nudge?.compressibleRanges) is empty because the kernel's computeProtectedRefs (soft recent zone: preserveRecentMessages + preserveRecentTokens + last user message) swallows all foldable content, and src/preflight.ts:264 then gives up with kind: "exhausted" → src/server.ts:1853 → permanent 502.

    Plan (per the issue's suggested fix):

    1. On the exhausted path, re-run the pipeline with the soft zone relaxed (preserveRecentMessages: 0 / preserveRecentTokens: 0), still honoring hard protectedTools, and fold oldest-first within the zone.
    2. If still nothing, try the kernel's truncateLargeToolOutputs as a last resort.
    3. If both fail, make the 502 actionable (name the protected zone + sizes + operator options).

    Working on it now — will report back with a PR.

  2. ranxianglei commented on Aug 29, 2026

    @ranxianglei
    OwnerAuthor

    🤖 ework agent · vllm-qwen/qwen3.8-27b

    [bot] Fixed — PR: #337

    Root cause (two independent brickers):

    1. The server pre-check in preflightCompressIfNeeded failed fast on the normal-config compressibleRanges — which exclude the soft-protected recent zone — before preflightCompress ever ran. So a payload whose only foldable content is inside the zone never got a chance.
    2. Even when preflight ran, it had no recovery once nothing was foldable outside the zone: it just set an exhausted failure → 502, forever.

    The fix:

    • Soft-zone relaxation (src/preflight.ts): when no range is foldable outside the soft zone and the payload's own estimate still overflows, preflight relaxes the soft zone to zero (preserveRecentMessages/preserveRecentTokens → 0) and retries, folding oldest-first within it. The hard protectedTools exclusion is computed independently and still applies. Only when truly nothing is foldable does it fail fast — now with an actionable error ("Raise the model context window or restart the session to recover") instead of the opaque "no compressible ranges remain".
    • Gate on the real payload, not the floor: the relaxation is gated on result.payloadEstimate, not currentTokens — the latter is floored by a possibly-stale lastInputTokens from a prior model, which would fold protected content even when the real payload already fits. (This regression was caught by the existing bili 模式切换模型会报错上下文满 #247 model-switch test during development.)
    • Suppress the emergency-truncate node in the preflight's per-round pipeline (window inflated so usage < truncate.threshold; compressible ranges derive from the protected zone, not usage). Without this, the node trims the very tool output overflowing the window, making the fit check undershoot and break before the soft zone is relaxed.
    • Forward/fail on the payload actually forwarded (src/server.ts): re-measure after the rebuild instead of trusting result.payloadEstimate, which the relaxed-zone processTurn trims more than the normal-config prepare does (could turn a guaranteed-400 forward into a false "fits").

    Tests: new e2e for the #330 scenario (over-window payload, only foldable content in the protected zone → relaxed, folded, forwarded 200); the "truly incompressible" e2e is now a hard-protected tool result (502 + actionable remedy, no summarization call, no forward). Full suite 746/746, typecheck + build clean.

    On the issue's second suggested lever (kernel truncateLargeToolOutputs): it's stateless/non-persistent and only touches tool-results (this repro is a large user message), so it's largely redundant with the soft-zone relaxation — noted as a possible follow-up in the PR rather than bundled here.

  3. added 3 commits that reference this issue on Sep 5, 2026
  4. added a commit that references this issue on Sep 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions