Skip to content

arm emergency shrink on upstream failure to break relay-5xx deadlock #604

Closed

Description

@ranxianglei

Problem

When the session is over budget and the upstream is a relay that answers overflow with a hard 5xx (instead of an in-band error), the request fails; recovery relies on the client retrying into a proxy that nudges. A client that just surfaces the error leaves a stuck, never-compressed session.

Partially mitigated since: stream retry (#440/#444), preflight hold (#571), learned-window self-correction (#570/#572) — but the "on upstream failure, force compression before the next attempt" mechanism itself is still absent from master.

Impact

Dead sessions on 5xx-on-overflow relays until manual intervention.

Suggested fix

On upstream failure with an over-budget session, arm the emergency compression path (truncate/emergency threshold already exist in compress-settings) so the next client retry lands on a foldable view. PR #229 carries a v0.1.7x-era implementation + e2e (tests/e2e-relay-5xx-selfheal.test.ts) as reference; needs rework against current server.ts.

Activity

  1. ranxianglei commented on Sep 7, 2026

    @ranxianglei
    OwnerAuthor

    🤖 ework agent · qwen3.8-27b

    [bot] 🏷 Picked this up. Triage done, implementation in progress — findings below, PR to follow in this thread.

    1. Reproduced/verified the gap (it's real, and it's exactly where you said). In forward() on master:

    • non-2xx path (src/server.ts:3155-3263): only inspectContextOverflow() is consulted, and by design it matches 400/413 only (src/util.ts:213). A relay 5xx falls through to log + verbatim passthrough with zero session-state change.
    • network-level failure (src/server.ts:3055-3058): throw → generic 502, again no state change.

    So after any failed turn, session.stats.lastInputTokens stays frozen at the last successful usage report — and it is the single input to every usage-driven trigger (preflight trigger floor max(lastInputTokens, payloadEstimate) at server.ts:2765, nudge bands, and the kernel's emergency band: acp-kernel 0.0.54 defaults truncate.threshold = 0.95, nudge.emergencyThresholdPct = 0.95). A client that retries verbatim re-sends the identical payload into the identical rejection forever. Confirmed: no arming mechanism exists anywhere on master; the only armed-channel write outside prepare is the overflow-400 path (server.ts:3231).

    2. Layer judgment. Your description is the root cause, not a symptom: failed turns never produce a usage report, so the only signal that drives compression goes stale. The listed mitigations all key off something a generic unsigned 5xx doesn't provide — #570/#572 learn from an overflow signature in the body, #440/#444 retry the same stream, preflight (#330/#337) only runs when its own estimate already says over-window. Your suggested fix hits the core: it injects a locally-computed lower bound into the same armed channel the overflow-400 path already uses (lastInputTokens = max(...) + markDirty), which puts the next prepare() inside the kernel's emergency band so processTurn truncates large tool results server-side without model cooperation. The next successful usage report overwrites the armed value (raise-only semantics).

    3. Rework notes vs PR #229 (v0.1.7x era — its e2e premise no longer holds): master's preflight now intercepts payloads whose local estimate exceeds the window before forward (fold, or fail-fast 502 per #301/#306). So the modern deadlock shape is narrower: the payload fits bili's own estimate but not the relay's hidden tolerance / near-window skew — exactly where an unsigned 5xx gives no learning signal. The e2e encodes that shape deterministically (fixture sits just under bili's window, above the mock relay's tolerance; arm crosses the 0.95 emergency band; next retry truncates + recovers with the [truncated for context space] marker). Two fire points, both raise-only: upstream status >= 500 (after the overflow block — inspectContextOverflow can't match 5xx, so no double-arm) and network-level fetch failure (non-GET/HEAD, before the throw; timeout aborts included — that's the ~31s relay-timeout case from the original repro). 4xx deliberately excluded (must not distort the usage signal), covered by a must-not-arm test.

    Known residual (documenting, not fixing here): if the relay's true tolerance is far below bili's configured/learned window and the error body carries no window number, an estimate-based arm can't cross any threshold — an unsigned 5xx fundamentally cannot reveal the relay's limit. That case still needs a signed error or manual intervention. (The alternative — arming to max(est, limit), i.e. treating every 5xx as "too big" — would force the truncate band unconditionally but fabricates a 100%-usage claim from possibly-transient blips; the reference design's raise-only lower bound is the safer default. Happy to switch if you disagree.)

    Pre-flight will be typecheck + full npm test + build, then the PR linked here.

  2. ranxianglei commented on Sep 7, 2026

    @ranxianglei
    OwnerAuthor

    🤖 ework agent · qwen3.8-27b

    [bot] 🏷 Done — PR: #611

    Triage verification

    Gap confirmed on master. In src/server.ts forward(), both failure paths that end without a usage report had no arming mechanism:

    • Upstream non-2xx (~line 3155): the !upstream.ok block only handles overflow signatures via inspectContextOverflow() (which matches 400/413 only — verified in src/util.ts:211) and learns windows from those. A generic relay 5xx fell through to logging + verbatim passthrough with the session untouched.
    • Network-level failure (~line 3055): the fetch catch recorded the connection error and threw — session untouched.

    Layering. Root cause: a failed turn produces no usage report, so session.stats.lastInputTokens — the input to every usage-driven trigger (preflight floor, nudge bands, the kernel's emergency nudge + tool-result truncate at truncate.threshold) — stays frozen at the last successful turn's value. The stuck-session symptom you describe is exactly what follows: every verbatim client retry re-sends the identical payload into the identical rejection. Your suggested fix hits the core, not the symptom: it extends the same raise-only arming already used by the overflow-400 path (#570/#572) to the two unsigned-failure classes. Raising the value is safe because it's a lower bound (compress earlier, never later), the kernel no-ops below its thresholds, and the next real usage report overwrites it.

    Duplicate screening: not a duplicate — #229 is the v0.1.7x-era reference implementation this issue explicitly cites as needing rework (kept open as spec); #470 (plugin-mode truncation tracking, closed via #576) and #301/#571/#572 are adjacent but distinct mechanisms.

    What shipped (PR #611)

    • armFailureShrink(prepared, log, reason) in src/server.ts — raises lastInputTokens to estimateTokensFast(prepared.body) (local estimate of the wire body we just sent — strictly better than fix: arm emergency shrink on upstream failure to break relay-5xx deadlock (#11) #229's raw client body), markDirty because both call sites return before forward()'s trailing save.
    • Fire point 1: upstream status >= 500, sibling of the overflow gate (arms even when the error-body stream broke).
    • Fire point 2: network-level fetch failure (socket reset / timeout abort), non-GET/HEAD only — the ~31s relay-timeout case from your original repro IS a timeout abort, so it's covered.
    • New tests/e2e-relay-5xx-selfheal.test.ts (real proxy + mock relay with a hidden context tolerance): ① 500 → passthrough verbatim → armed (value lands in the 95–100% window band, state persisted) → verbatim retry gets 200 with server-side-truncated tool results ([truncated for context space]) that fit the tolerance → final lastInputTokens === 5000 proving the real usage report overwrote the estimate; ② network destroy → bili 502 → same recovery; ③ 401 must NOT arm (usage signal undistorted).
    • Pre-flight: typecheck ✓ · full suite 1144/1145 (the one failure is a pre-existing environmental launcher.test.ts codex-path assertion, identical on clean master — verified via stash) · build ✓ · e2e-codex zero-token preflight ✓.

    Known residual

    An unsigned 5xx reveals nothing about the relay's actual limit. If the payload fits bili's configured window but exceeds the relay's hidden tolerance far below it, the armed estimate stays under bili's limits and no trigger fires — that case still needs the client to surface the error or a future learned-limit signal from the relay. Documented in the PR body too.

  3. added a commit that references this issue on Sep 7, 2026
  4. added 2 commits that reference this issue on Sep 15, 2026
  5. added a commit that references this issue on Oct 18, 2026
  6. added a commit that references this issue on Oct 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions