Skip to content

[PR] fix: propagate upstream error-body failures without replay - #6134

Closed
Afloat16 wants to merge 1 commit into
ranxianglei:masterfrom
Afloat16:2026-11-10_fix-retry-body-errors
Closed

Afloat16 wants to merge 1 commit into
ranxianglei:masterfrom
Afloat16:2026-11-10_fix-retry-body-errors

Conversation

@Afloat16

Copy link
Copy Markdown
Contributor

Problem

fetchWithRetry replaces every HTTP error-body read failure with "upstream error". For HTTP 503, body timeouts and client cancellation become retryable HTTP verdicts, replaying requests and hiding the transport cause.

Changes

Read the complete error body before HTTP retry classification. Propagate failed reads unchanged and clear the request watchdog in finally. Complete transient error responses retain bounded retries.

Validation

  • Three regressions fail on unchanged source: body timeout at HTTP 400 and 503, and client cancellation while reading a 503 body.
  • 14 targeted tests passed in retry-error-body and upstream-fail, including successful HTTP retry, timeout/abort classification, transport recovery and zero leaked watchdogs.
  • Final-code typecheck, npm build with dist checks, and git diff --check passed. The complete suite is left to CI.

Checked prior retry work #190/#1351/#1454/#1693/#5947 and Afloat16 history. This concerns errors after response headers arrive.

Base: e1e33a9.

Closes #6133

@ranxianglei ranxianglei changed the title fix: propagate upstream error-body failures without replay [PR] fix: propagate upstream error-body failures without replay Nov 10, 2026
@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重: 不重复。 Searched open+closed issues/PRs for the error-body-swallow mechanism — the only items are #6133 (source issue) and this PR #6134. The prior retry work you cite (#190/#1351/#1454/#1693/#5947) are different failure domains (pre-response transport deaths / the original retry policy), not this post-header body-read path, so they're related infrastructure, not duplicates of this fix.

Review — PR #6134 (external contributor)

What I verified (evidence, not "should work"):

  • Read the full fetchWithRetry (src/fetch-util.ts:529) and both callers; traced the exact behavior change.
  • Ran the targeted suite locally: tests/retry-error-body.test.ts → 4/4 pass on the new code; re-ran against the old fetch-util.ts → exactly 3 fail / 1 pass, matching your claim (400 body-timeout, 503 body-timeout, abort-during-503-body fail on old source; the "complete transient still retries" case passes on both). The test has real teeth.
  • npm run typecheck clean (exit 0). Live CI via mirror cache at review time: 0 failed, 3 queued (mysql-contract / docs-sync / architecture-docs); all completed guards green.
  • Confirmed the PR's imported test seams already exist on master (_liveUpstreamTimersForTest, _resetFetchUtilForTest, ReplayRetryInfo, withFetchTransport) — the test loads cleanly.
  • Swept the whole repo for the same swallow pattern: src/fetch-util.ts:552 is the only .text().catch(() => "upstream error"); src/preflight.ts:1489 already does a bare await …text() (no catch), so fetchWithRetry was the sole buggy site. No sibling instances left un-fixed.

① Direction (original issue #6133): Correct root cause, correctly layered. The defect is that a body-read failure after headers arrive is disguised as a complete transient HTTP error, so 503 gets replayed and the transport cause is lost. Your fix removes the producer (the blanket catch) rather than masking the symptom downstream. Notably the function's own doc comment (src/fetch-util.ts:527: "headers/body timeouts and aborts still propagate unchanged — NOT retried") is what the old code violated — the fix restores the documented contract. The direction is right.

② Regression review — pre-existing behaviors I checked, with verdicts:

  • Success path (response.ok, :551): unchanged — returns before the timer block; caller owns the timer. ✔ preserved
  • Complete error body + transient status (503): unchanged — still classified by isTransientUpstreamError, still bounded-retried. Locked by test 3. ✔ preserved
  • Pre-response network-failure classification (inner catch, classifyUpstreamFailure, :541–550): untouched. ✔ preserved
  • Both call sites rethrow non-UpstreamHttpError throws (src/loop/core.ts:629, src/compress-loop-responses.ts:234): the newly-propagated body-timeout/AbortError now rides these existing rethrow paths — consistent with how non-fail-fast network errors already propagate. No new crash surface. ✔ no regression
  • Regression count: 0. One intended change, disclosed below.

③ Code review: Minimal and correct. try { errText = await response.text() } finally { clearTimer() } guarantees the watchdog is cleared on every path (success or read-failure) while letting the raw transport error escape instead of being reclassified. Timer is cleared exactly once per attempt (no double-clear vs. the old code). Naming/style match the file; no dead code, no magic values beyond the meaningful [400, 503].

④ Test validity: Real assertions with teeth (identity check error === failure, sends === 1, retries === [], and live === 0 watchdog-leak guard in afterEach). Covers both directions required by gate discipline (#3617): propagates-when-it-should AND retries-when-it-should. Import convention matches siblings. Good.

⑤/⑤b Diff cleanliness & code-quality hard bar: 2 files, +88/−2, single commit, every line on-topic; no unrelated churn, no whitespace noise, no lock/generated changes. Passes the external-submission strict bar.

Intended behavior change (must flag — Deliverables #4):

  • Old: body-read failure (timeout/abort) on an HTTP error response → swallowed to "upstream error" → for a transient status (503) the request is replayed up to maxAttempts; otherwise UpstreamHttpError(status, "upstream error"). Log detail = misleading "upstream error".
  • New: body-read failure → the raw transport error (UND_ERR_BODY_TIMEOUT / AbortError / …) propagates immediately, no replay, watchdog cleared in finally. Log detail reflects the real cause.
  • Complete error bodies are byte-for-byte unchanged in handling. This is the bugfix, not drift.

结构化八问

  1. 是什么问题: fetchWithRetry 读上游错误响应体时,把任何 .text() 读失败(UND_ERR_BODY_TIMEOUT、客户端 abort)一律吞成字符串 "upstream error"。对 503 这类瞬时码,吞掉后仍走瞬时重试 → 多余重放请求,且丢失真实传输层原因。
  2. 什么原因导致的: 根因在 errText = await result.response.text().catch(() => "upstream error")(src/fetch-util.ts:552)。无差别 catch 把「响应头已到、但读体失败」这一传输层故障伪装成「拿到完整瞬时 HTTP 错误」,于是被 isTransientUpstreamError(503,…) 判为可重试。这与该函数自己的文档契约(:527 注释「headers/body timeouts and aborts still propagate unchanged — NOT retried」)直接矛盾——是代码没兑现契约,不是契约错了。
  3. 影响范围与严重程度: 触发条件 = acp-loop / responses-compress-loop 两车道里上游返回非 ok 状态码且读响应体失败。后果:请求被多余重放(最多 maxAttempts 次,每次可能吃满 idle-timeout 预算)+ 日志 detail 变成误导性的 "upstream error"。p2:特定条件下的行为缺陷 + 误导性诊断,能用、不卸载级;正常完整响应不受影响。
  4. 怎么修复的: 去掉无差别 catch,改为 try{ errText = await text(); } finally{ clearTimer(); } —— 读体失败让原始传输错误直接上抛、finally 保证 watchdog 一定清理,不再进入瞬时分类;完整读到的错误体仍按原逻辑判定与有界重试。属「修生产者」,不在错误的层把传输失败改写成 HTTP 结论。
  5. 是否有回归问题: 逐项审过既有行为——成功路径(不变)、完整错误体的瞬时重试(test 3 钉住,不变)、预响应网络失败分类(未触碰,不变)、两个调用方的非 UpstreamHttpError rethrow(新上抛的 body-timeout/abort 复用既有 rethrow 路径,一致)。回归 0 处;唯一有意变更即上文披露的 body-read-failure 处理。
  6. 是否破坏缓存: 不涉及缓存/重试游标/去重键。反而减少了本不该发生的重放,避免对上游的重复投递;与 prefix-cache 无关。
  7. 是否改了设计方向: 不改架构/边界/数据流。是把实现拉回该函数已声明的既有契约(propagate body timeouts & aborts),bugfix 而非转向。
  8. 是否引入了新的配置: 不涉及新增 env/config/默认值。复用既有 BILI_REPLAY_RETRY_MAX / BILI_REPLAY_RETRY_BASE_MS 预算。

判定: 可合。 修法正确、测试有牙、diff 干净、方向对路。唯一动作项:PR 落后当前 master ~52 个提交(但 master 未触碰 src/fetch-util.ts,GitHub 已标 mergeable,无冲突)——合并前 routine rebase 到 master 即可。按规则我不执行合并,请 owner 自行合并: #6134

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Status update — my full structured review is already up in this thread (verdict unchanged); this is a live CI re-check that found the picture moved since my first pass.

CI (verified live against PR head 255ab4e1, single commit — head has not moved since creation): the only check-run on this head started 15:24Z Nov 10 and was cancelled mid-flight at ~15:52Z Nov 10. Final tally: 9 cancelled / 10 success / 11 skipped / 0 failed. The cancellation hit every in-flight job at once (~15:52:4x): all 6 ubuntu-latest test shards + 2 windows shards (22×2, 22×5) + red-signal. What did go green: windows 22×1/×3/×4, static-gates (both OS), mysql-contract, docs-sync, architecture-docs, kernel-guard. The 11 skips are the gated zero-token e2e lanes. No newer run exists and the head is unchanged, so nothing will flip back to green on its own.

Net: the code-review verdict stands — 可合 (see the review above) — but this head currently has no clean full-suite pass: the ubuntu test matrix never completed. Two prerequisites before merge:

  1. Routine rebase onto current master (PR is behind; master has not touched src/fetch-util.ts, so it merges cleanly — GitHub reports mergeable).
  2. Trigger a fresh full CI run so the ubuntu test shards actually finish green.

If you want the mid-flight cancellation itself root-caused (manual cancel vs runner issue), that's CI-infra territory — a line in the CI coordination thread #4599 beats chasing it inside this PR. I did not chase it here; it does not affect the code-review verdict.

Per repo rules I don't merge — owner merges when ready: #6134

@Afloat16
Afloat16 force-pushed the 2026-11-10_fix-retry-body-errors branch from 255ab4e to 70f175b Compare November 11, 2026 04:37
@Afloat16

Copy link
Copy Markdown
Contributor Author

Rebased the unchanged two-file fix onto current master (33104b1) in 70f175b, as requested. The resulting tree matches the clean merge result.

Validation on the rebased tree: all 14 tests in retry-error-body and upstream-fail pass, plus npm run typecheck and npm run build. Fresh CI was triggered: https://github.com/ranxianglei/billion-context/actions/runs/42960469014 . The core test shards have started executing; the Windows 24 full-matrix lane still needs the maintainer-controlled ci:full run.

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Verified your rebase live — thanks. Current state:

Rebase (checked locally against git): head is now 70f175b, sitting directly on the current master tip (33104b1) — 0 commits behind. The rebased commit is the same two-file change I reviewed (src/fetch-util.ts + tests/retry-error-body.test.ts, +88/−2, identical message) — the rebase was clean/mechanical, no content drift or scope creep. (Note: the GitHub API momentarily reported mergeable_state: behind, but that's async lag right after a force-push; local git confirms it's clean against master.)

Fresh CI (run 42960469014, read live): fully terminal and green — 18 success / 12 skipped / 0 failed / 0 cancelled / 0 pending. All 11 test-matrix shards actually ran and passed (ubuntu 22×1–3, ubuntu 24×1–3, windows 22×1–5); static-gates (both OS), mysql-contract, docs-coverage, gate, test-import-graph, unused-exports all green. The 12 skips are the default-gated zero-token e2e lanes + red-signal. The mid-flight cancellation from the first run did not recur.

On your Windows-24 full-matrix note: understood — that lane (and red-signal) sit behind the maintainer-controlled ci:full dispatch, outside the standard per-PR gate. It doesn't block the code-review verdict; flagging it here so you/the owner can trigger ci:full before merge if you want that extra coverage.

Verdict: unchanged — 可合. Both prerequisites from my last note are now satisfied (clean rebase onto current master + fresh CI green). Ready for owner merge whenever. Per repo rules I don't merge — owner merges when ready: #6134

结构化八问(本轮增量,代码级答案与上轮一致)

  1. 是什么问题: 不变——body-read 失败被吞成 "upstream error",导致 503 被重放、真实传输原因丢失。
  2. 什么原因导致的: 不变——无差别 .catch(() => "upstream error")(src/fetch-util.ts)。
  3. 影响范围与严重程度: 不变——p2(特定条件下的行为缺陷 + 误导性诊断)。
  4. 怎么修复的: 不变——去掉 catch,改 try{…}finally{ clearTimer(); } 让原始传输错误上抛、watchdog 必清。
  5. 是否有回归问题: 不变——回归 0 处;唯一有意变更即 body-read-failure 处理。本轮新增事实:rebase 后 diff 逐字节未变(+88/−2 两文件),既有行为判定全部沿用。
  6. 是否破坏缓存: 不涉及(同前)。
  7. 是否改了设计方向: 不涉及(同前)。
  8. 是否引入了新的配置: 不涉及(同前)。

合并前置项现状: ① rebase → ✅ 已干净落在当前 master;② 新鲜 CI → ✅ 标准套件全绿。仅剩 owner 合并(可选:触发 ci:full 补 Windows-24 全矩阵覆盖)。

@ranxianglei

Copy link
Copy Markdown
Owner

Landed via batch PR #6439 (merged) — closing as superseded.

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Confirmed live: #6134 is closed (superseded, not merged directly) and batch PR #6439 is merged — the fix landed there with the other Afloat16 fixes. Nothing further needed on my side; no action taken on the closure (owner-side). Thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] HTTP error-body timeouts are swallowed and replayed as transient HTTP failures

2 participants