Skip to content

[PR] fix: bound compression replay error bodies - #7203

Open
Afloat16 wants to merge 1 commit into
ranxianglei:masterfrom
Afloat16:2026-11-18_bound-replay-error-bodies
Open

Afloat16 wants to merge 1 commit into
ranxianglei:masterfrom
Afloat16:2026-11-18_bound-replay-error-bodies

Conversation

@Afloat16

Copy link
Copy Markdown
Contributor

Fixes #7200.

Problem

Compression replay requests read every rejected response with response.text() before deciding whether to retry. A real HTTP fixture retains all 8 MiB in either UpstreamHttpError.body or the retry callback. A continuously extended error response can also postpone the decision indefinitely because incoming chunks rearm the idle watchdog.

Changes

Read at most 1 MiB of response-body bytes into the replay diagnostic. Once overflow is confirmed, cancel the unread tail without awaiting cancellation, and prefix the diagnostic with an explicit truncation marker. Streaming UTF-8 decoding preserves split characters; a code point intentionally cut at the byte boundary is left out rather than replaced.

Complete error bodies at or below the limit keep their existing decoding. Read failures and aborts retain their original errors, and the existing finally still clears the watchdog. Success streams, the main passthrough transport, attempt limits/backoff, and Retry-After handling are unchanged.

This intentionally stops inspecting prose retry markers beyond the byte budget. A marked prefix is not presented as a complete JSON envelope. The limit concerns retained source bytes, not a claim about transport chunk size or total process memory. No config, kernel, or persistence change.

Validation

  • 22 new regressions: 13 fail / 9 pass on unchanged master; all pass after the change.
  • 109 targeted tests pass across replay failures, upstream diagnostics, transport retries, streaming-loop 429 observation, and the non-streaming Anthropic/Responses loops.
  • Same 8 MiB HTTP fixture before/after: the terminal 400 diagnostic and the 503 retry diagnostic each shrink from 8,388,608 to 1,048,628 bytes (1 MiB ASCII prefix plus a 52-byte marker). Request counts remain one and two respectively; both runs leave zero live watchdogs.
  • Real HTTP integration: the Anthropic acp_status loop receives a 503 body exceeding the cap with no EOF. The original waits until the test's external abort; the fix cancels that body, retries, and returns the supplier's final answer. Both attempts carry the identical executed tool receipt.
  • Boundary, multibyte/BOM/invalid UTF-8, explicit truncation, cancellation rejection/non-settlement, read-failure identity, finite attempt budget, and large successful response controls.
  • npm run typecheck, npm run build, import-graph check, and whitespace check pass on current master a446a3c1.

The Codex preflight finds the CLI and built artifact, but the configured real-model upstream on port 8199 is unreachable. The required four real-model phases have not run; this remains a before-merge gate. Full regression is the remote CI verdict on the pushed head.

Duplicate check

All 270 fork heads and 279 upstream file-history commits were checked. The earlier error-identity changes in #6134/#6159 and declaration-parser bound in #3610 do not cap this read. The body cleanup in #7171 belongs to makeContinuationRefetch, which discards a response entirely; this change bounds diagnostic consumption in a different replay path.

@ranxianglei ranxianglei changed the title fix: bound compression replay error bodies [PR] fix: bound compression replay error bodies Nov 18, 2026
@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

Independent review — I re-ran the verification myself rather than re-reading the issue's conclusions.

What I verified

  • Bug reproduced on unchanged master (a446a3c): ran the PR's new test file against the base — exactly 13 fail / 9 pass, matching the PR's claim.
  • Diff: only src/fetch-util.ts (+33/−2) and new tests/retry-error-body-budget.test.ts (+280). Single commit, clean message, no unrelated lines, git diff --check clean.
  • Traced every consumer of the bounded body:
    • isTransientUpstreamError: 429/5xx are status-based (body unused); the 4xx prose markers are now scanned in the first 1 MiB only — the documented behavior change.
    • bodyDeclaresRetryable: bodies over 65,536 chars were already rejected; a truncated body is always over 65,536 chars → verdict unchanged.
    • throttle429ModelScope → keyringModel429Scope: MAX_KEYRING_ERROR_BODY_BYTES = 16 KiB already rejects larger bodies → verdict unchanged.
    • Thinking-signature regexes in src/loop/core.ts (cannot be modified / must be passed back): those provider bodies are small JSON envelopes, never realistically truncated.
    • The marker: 52 bytes, contains no transient marker, is not JSON; host-log/internal only, not model-visible → no model-facing text change.
  • UTF-8 semantics: bodies ≤ 1 MiB are byte-identical to Response.text(); the exact cap completes without the marker; a code point cut at the boundary is dropped, not replaced; read failures preserve error identity (test asserts error === failure).
  • Watchdog: finally { result.clearTimer(); } unchanged; the real-HTTP test asserts the server observes res 'close' (the client actually cancelled) and afterEach asserts zero live timers.
  • Adjacent suites on head: 22/22 new + 37/37 retry/error suites + 22/22 loop suites. typecheck + build + import-graph pass.
  • CI on pushed head c94f95a: 36 success / 12 skipped / 0 failed — fully green (verified the head is unchanged since).

Behavior changes (disclosed, old → new)

  1. Error bodies over 1 MiB: full body retained → 1 MiB prefix + 52-byte truncation marker (the fix itself).
  2. 4xx prose-marker retry scan: full body → first 1 MiB only; a marker past 1 MiB now fails fast instead of retrying (documented in the issue + PR, test-pinned).
  3. Log lines for oversized errors: the first 120/200 chars now show the marker.
  4. JSON retry declarations and keyring 429 scope: no behavior change (already capped below 1 MiB).

Non-blocking follow-up found in review
src/preflight.ts:1497 has the same unbounded read in the preflight summary retry loop (direct fetchWithTimeout). Same defect class: full drain while the watchdog is armed, body retained in UpstreamHttpError.body, plus a file dump via dumpSummaryRejection. Out of scope for this PR (the issue deliberately scopes to fetchWithRetry; preflight is a load-bearing hot path deserving its own diff + tests). Recorded as a checklist item on #7200 with a proposed fix (branch on response.ok: full text() on success, bounded read on error reusing the exported helper). Recommend a small follow-up PR after this one merges.

§7.4 gate check: all 8 conditions hold — single module, regression test reproduces the bug (13 fails on master), green on the rebased head, no config/persistence/wire/kernel change, pure fix:, clean diff, Fixes #7200, no load-bearing infra. Qualifies for auto-merge; the merge itself is human-only.

Remaining gate the author lists: the four real-model e2e phases have not run (configured upstream on port 8199 unreachable). Per the PR body this is a before-merge gate; the full-suite verdict is the remote CI, which is green.

8问(结构化结论)

  1. 是什么问题:fetchWithRetry 对每个被拒响应先全文读取 response.text() 再决定重试——超大错误体被完整保留在内存(UpstreamHttpError.body / 重试 detail),且慢速滴灌的错误体会不断重置空闲看门狗,无限推迟重试决策。
  2. 什么原因导致的:重试判定前对错误体做了无界全量读取——读取本身没有字节预算,内存保留与看门狗重置都是这个无界读取的派生后果。
  3. 影响范围与严重程度:所有走 fetchWithRetry 的压缩回放路径(流式 loop、Anthropic/Responses 压缩 loop)在上游返回超大错误体时:内存无界保留 + 决策被无限推迟。特定条件下有问题(超大错误体),非普遍性故障;p2 合理。
  4. 怎么修复的:新增 readReplayErrorBody——手动 reader 循环,最多保留 1 MiB 源字节,增量 UTF-8 解码;确认溢出后 fire-and-forget 取消未读尾部(不等待取消,因为取消可能挂起在未消费的 tee 兄弟流上),诊断文本前加显式截断标记;finally 释放锁;fetchWithRetry 的 finally 清看门狗逻辑不变。
  5. 是否有回归问题:审过的既有行为——≤1 MiB 错误体逐字节不变(与 Response.text() 相同);精确 1 MiB 不带标记;JSON 重试声明(>65,536 字符本已拒绝)与 keyring 429 scope(>16 KiB 本已拒绝)判决不变;thinking-signature 错误检测正则针对的小体量 provider 错误体不受影响;成功流、主透传、尝试次数/退避、Retry-After 均未触碰。有意的行为变更 3 处已在上文披露(截断保留、4xx 散文标记只扫前 1 MiB、超大错误日志行首现标记)。
  6. 是否破坏缓存:不涉及——错误路径诊断保留,不触碰前缀缓存、重试游标、去重键;对一切现实大小的错误体,重试判决不变。
  7. 是否改了设计方向:不涉及——无架构/边界/数据流变更;无配置、kernel、持久化变更。
  8. 是否引入了新的配置:不涉及——无新 env/config/默认值。

Verdict: mergeable. All §7.4 auto-merge conditions hold; CI green on the pushed head; the only remaining gate is the real-model 4-phase e2e the author could not run (upstream unreachable) — run it (or accept the CI verdict + the new real-HTTP integration test) before merge. The src/preflight.ts:1497 instance is tracked on #7200, non-blocking.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Compression replay retries consume unbounded upstream error bodies

2 participants