Skip to content

fix: arm emergency shrink on upstream 5xx/network failure (#604) - #611

Merged
ranxianglei merged 1 commit into
masterfrom
2026-09-07_arm-failure-shrink
Sep 7, 2026
Merged

ranxianglei merged 1 commit into
masterfrom
2026-09-07_arm-failure-shrink

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

What

Fixes #604 — rework of PR #229 (v0.1.7x-era implementation + e2e) against current server.ts.

A failed turn produces no usage report, so session.stats.lastInputTokens — the input to every usage-driven trigger (preflight floor, nudge bands, the kernel's emergency nudge + tool-result truncate at truncate.threshold) — stays frozen at the last successful turn's value. On a relay that answers overflow with a hard 5xx (no in-band error, no usage), every verbatim client retry re-sends the identical payload into the identical rejection forever.

This adds armFailureShrink() in src/server.ts, fired from the two failure paths that never report usage:

  1. Upstream 5xx — new branch in the !upstream.ok path, sibling of the overflow-400 gate (fires even when the error body stream broke). inspectContextOverflow only matches 400/413, so the learned-window path cannot have handled these responses; no double-arm possible.
  2. Network-level failure — socket reset / timeout abort in the fetch catch (non-GET/HEAD only), armed before the throw surfaces as bili's 502. The ~31s relay-timeout case from the original repro IS a timeout abort, so it must be covered.

Both raise lastInputTokens to estimateTokensFast(prepared.body) — a local estimate of the wire body we just sent (strictly better than PR #229's raw client body). The value RAISES only (a lower bound → compress earlier, never later); the kernel no-ops below its thresholds; the next successful usage report overwrites it. markDirty is required because both call sites return before forward()'s trailing save.

On the next client retry, prepare() lands in the kernel's emergency band and truncates large tool results server-side ([truncated for context space]) without model cooperation — the next attempt lands on a foldable view.

Test

New tests/e2e-relay-5xx-selfheal.test.ts (3 tests, real proxy server + mock relay with a hidden context tolerance):

  1. Relay 500 on oversized request → passed through verbatim → session armed (lastInputTokens ≥ ceil(sentBytes/8), inside the 95%–100% window band) → state saved → verbatim retry gets 200 with truncated tool results that fit the relay's tolerance → final lastInputTokens === 5000 (real usage report overwrote the armed estimate).
  2. Network-level destroy → bili 502 upstream request failed → same arm → retry recovers via emergency truncate.
  3. 401 (auth failure) must NOT arm — lastInputTokens stays 0; 4xx must not distort the usage signal.

Calibration note: the relay tolerance band had to account for the ~5.5k-char emergency-nudge message the kernel appends during recovery — the fixture constants document the measured (sentLen2/4, sentLen1/4) band explicitly.

Pre-flight

  • npm run typecheck ✓
  • npm test ✓ 1144/1145 — the single failure is tests/launcher.test.ts (expects bare codex command, sandbox resolves /usr/bin/codex): pre-existing environmental, verified identical on clean master via stash
  • npm run build ✓
  • E2E_CHECK=1 tests/e2e/e2e-codex.test.ts ✓ (zero-token preflight; full ACP_TEST_E2E run needs a live upstream)

Known residual (documented in issue reply)

An unsigned 5xx reveals nothing about the relay's actual limit. If the payload fits bili's configured window but exceeds the relay's hidden tolerance far below it, the armed estimate stays under bili's limits and no trigger fires — that case still needs the client to surface the error or a future learned-limit signal.

@github-actions

github-actions Bot commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-09-07_arm-failure-shrink (ae23937)

Option A — Install from npm PR tag (recommended)

npm install -g billion-context@pr-611

Each push to this PR publishes a new version under the pr-611 npm tag.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr611.tgz
npm install -g package

This comment is automatically updated on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 ework agent · qwen3.8-27b

[bot] 🏷 Independent review of PR #611 — LGTM, ready to merge (merge is human-only per policy; I won't merge).

I re-verified every claim against the code rather than taking the PR body at face value. Findings:

Mechanism is real, not accidental. In acp-kernel 0.0.54 the emergency-truncate pipeline node fires purely on tokenCount / modelContextLimit >= truncate.threshold (default 0.95) and calls truncateLargeToolOutputs(...); decideNudge.emergencyOverride keys off the same ratio. tokenCount is fed from session.stats.lastInputTokens (src/server.ts:1815), so raising that value genuinely puts the next prepare() into the emergency band and truncates large tool results server-side with no model cooperation — exactly what breaks the relay-5xx deadlock.

Both compression modes covered. prepareAnthropic (and the OpenAI/Responses siblings) set tokenCount = lastInputTokens → core.processTurn(...) → processedMessages = stripKernelSummaries(turn.messages) → body: JSON.stringify(rebuilt) built from those processed messages. pluginMode only gates tool injection (injectTools = ... && !pluginMode), never the rebuild — so the truncated tool results reach the wire in both proxy and plugin mode. This is the mode-pairing check §6 requires; it holds.

No double-arm. inspectContextOverflow (src/util.ts:213) returns early unless status === 400 || status === 413; the new branch requires >= 500. Mutually exclusive with the overflow/learned-window path, so one response can't be armed twice.

Overwrite, not permanent skew. On the next successful turn the reported-usage path does lastInputTokens = Math.max(0, total - credit) — it replaces the armed estimate rather than max-ing against it, so a transient blip can't leave a permanently inflated signal. estimateTokensFast is ceil(len/4); as a raise-only lower bound its error direction is benign (under-counts CJK, slightly over-counts escaped JSON — either way "compress earlier, never later"), and it's clobbered by real usage on recovery.

Save correctness. Both call sites return before forward()'s trailing save, so markDirty inside armFailureShrink is required and present; the 5xx test asserts scheduleSave fired ≥2× for the session.

Pre-flight (re-run here): typecheck ✓ · build ✓ · full suite 1144/1145 — the single failure is tests/launcher.test.ts (resolveClientCommand: codex/claude resolve to themselves), a pre-existing environmental assertion unrelated to this server.ts-only diff · new tests/e2e-relay-5xx-selfheal.test.ts 3/3 pass (5xx→arm→recover, network-destroy→arm→recover, 401→must-not-arm).

Non-blocking observations:

  • A transient/genuine 5xx on a session already near the window causes one extra round of emergency truncation on the next retry. Bounded and self-correcting (real usage overwrites the estimate), and strictly better than the infinite identical-retry deadlock — acceptable tradeoff.
  • Author-documented residual (agree it's out of scope): an unsigned 5xx where the payload fits bili's configured window but exceeds the relay's hidden tolerance far below it leaves the armed estimate under bili's own limits, so no trigger fires. Covering that needs a learned-limit signal from the relay; fine to ship with the disclosure already in the PR body.
  • Cosmetic nit (optional): the two call-site guards read differently (prepared && req.method !== … vs prepared?.session) — both correct; align for consistency if you like.

Recommendation: approve. Verified correct across both modes, no regression surface beyond the intended arming, pre-flight clean. I'm not merging — that's human-only.

PR: #611

@ranxianglei

Copy link
Copy Markdown
Owner Author

Review vs master c8c825b: ✅ merge-ready. Verified locally: typecheck ✓, tests/e2e-relay-5xx-selfheal.test.ts 3/3.

  • Only-raises semantics (est > s.stats.lastInputTokens) is the correct shape: a lower-bound estimate can compress earlier, never later; kernel no-ops below thresholds; next real usage report overwrites.
  • Both no-usage paths covered: network-level failure (catch branch, GET/HEAD excluded) and HTTP ≥500 — and the comment correctly notes inspectContextOverflow matches 400/413 only, so generic relay 5xx had NO self-heal path before this.
  • markDirty before the early returns — the arm survives restart ✓ (both call sites return before forward()'s trailing save).
  • [bug] bili dsh 长会话中 max_output_tokens 被压到 ≤200,主请求被误判为 side request 并绕过压缩 #546 interplay checked: side-request classification keys off budget watermarks, not lastInputTokens; armFailureShrink never touches budget → no misclassification. Worst case for a small session: a ~N-hundred-token title-gen body estimate arms a proportionally small value — harmless.
  • Body estimate uses the REBUILT wire body (post-fold, JSON-escaped → slight overestimate) — overestimate is the safe direction here.

(Closes #604 on merge)

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 ework agent · qwen3.8-27b

[bot] 🏷 Agreed — converges with my independent review; merge-ready. I separately re-checked the #546 interaction you flagged and confirm it: isSideRequest (src/server.ts:633) and sideRequestGuard (src/server.ts:697) classify purely off the request's output-budget field (max_tokens/max_completion_tokens/max_output_tokens) + tools presence (+ config/learned limits) — neither reads lastInputTokens. So armFailureShrink (which only mutates lastInputTokens) cannot flip a request's side/main classification. Clean.

One refinement to your "worst case is a small title-gen body" framing, verified in code:

Side requests DO reach both arm call sites with a populated prepared. At src/server.ts:1414-1424 a side request builds sidePrepared with body: bodyBuffer (the raw client body) and calls forward(...), so the network-catch guard (src/server.ts:3094, prepared && …) and the ≥500 branch (prepared?.session && …) both fire for a side request. And per the #554/#388 comments directly above (src/server.ts:1387-1391: "a full-history side request over the window…"), side requests can carry full-history bodies, not just N-hundred-token ones — that's exactly why sideRequestGuard estimates the raw body against the window at all.

Consequence: a 5xx / network failure on a full-history side request arms lastInputTokens to estimateTokensFast(fullBody) ≈ the real context size, not a small value. Is that harmful? No — it's roughly accurate (the side body is the conversation), it self-corrects on the next real usage report, and it does not violate #388's core invariant (no block/fold pollution: processedMessages stays [] and processTurn is skipped — only a usage stat is raised). So current behavior is benign.

Optional hardening (non-blocking): exclude side passthrough at both sites — if (prepared && !prepared.sidePassthrough && req.method !== "GET" && req.method !== "HEAD") / if (prepared?.session && !prepared.sidePassthrough && upstream.status >= 500). Rationale: (a) #604 is specifically the main-turn deadlock; a side request failing is a different shape (the client just re-issues title-gen); (b) #388's intent is that side requests don't influence the main session's state, and arming the main session's shrink off a side request's failure bends that; (c) a side body's purpose/shape differs from the main turn, so it's a weaker signal than the main turn's own payload. Two-line change — fine to land here or as a trivial follow-up, your call.

Net: agree with merge-ready. I won't merge (human-only per policy); on merge the PR closes #604.

@ranxianglei
ranxianglei merged commit bd89d58 into master Sep 7, 2026
6 checks passed
@ranxianglei
ranxianglei deleted the 2026-09-07_arm-failure-shrink branch October 15, 2026 05:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

arm emergency shrink on upstream failure to break relay-5xx deadlock

1 participant