Skip to content

[PR] fix: retry unreadable cache keepalive responses - #6735

Merged
ranxianglei merged 1 commit into
ranxianglei:masterfrom
Afloat16:2026-11-14_keepalive-body-failures
Nov 14, 2026
Merged

ranxianglei merged 1 commit into
ranxianglei:masterfrom
Afloat16:2026-11-14_keepalive-body-failures

Conversation

@Afloat16

Copy link
Copy Markdown
Contributor

Closes #6734.

Problem

An idle Anthropic cache ping can receive HTTP 200 headers and then fail while reading the response body. The current empty-string fallback turns that transport or decompression failure into a missing cache-usage result, drops the snapshot after one attempt, and stops future keepalive pings.

Change

Route response-body read failures through the existing three-consecutive-failure budget and normal cadence. A subsequent cache hit resets the streak. HTTP 429 still stops immediately, and complete responses with zero/missing cache usage keep their existing stop policy. The existing entry-ownership check also protects a replacement snapshot when an older body fails.

This does not add settings, change the ping request bytes, or alter the retry budget.

Validation

  • Eleven new tests use a real loopback HTTP server and native Fetch: truncated Content-Length and invalid gzip responses after HTTP 200 headers, successful recovery, three-failure exhaustion, unreadable 429/500 bodies, a response failing after rearming, and complete zero/missing/invalid-JSON usage controls.
  • On unchanged master a8bbb8d23108ba9148d09512927041364625d910: 4 new regressions fail and 7 controls pass. The existing two keepalive suites pass 45/45.
  • With this patch: 56 tests pass across the new suite and both existing suites.
  • npm run typecheck, npm run build, and git diff --check pass. Full regression status remains the remote CI result on this PR's head.

The tests do not call a live model provider; cadence is driven through the existing scheduler seam, while HTTP transfer and decoding are real.

Prior work

My #6078/#6077 protect a newly armed snapshot from a stale successful body; #6238/#6227 exclude tool data from TTL discovery. Their complete patches and the original #3390 were checked: this response-read failure path remained unchanged. Open/closed keepalive searches found no matching correction.

@ranxianglei ranxianglei changed the title fix: retry unreadable cache keepalive responses [PR] fix: retry unreadable cache keepalive responses Nov 14, 2026
@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷
(ai: vllm-qwen/qwen3.8-27b)

查重: 不重复. Searched open+closed issues/PRs for the keepalive family: #6737/#6739 (credential rotation leaves an unregistered snapshot held by its timer — different mechanism: timer retention, not response handling), #6650 (client-facing stream watchdog — different surface), #6078/#6077 and #6238/#6227 (closed; stale-body ownership and TTL-detection scope — adjacent facets, already fixed; this PR's rearm test pins the #6078 guard), #3390 (original feature that introduced the empty-string fallback). No existing PR addresses the response-body read-failure path.

Review of PR #6735 — fix: retry unreadable cache keepalive responses (Closes #6734)

I re-ran the author's claims independently instead of trusting the description:

  • Baseline: on unmodified master a8bbb8d2, the new suite fails 4/11 — exactly the four 200-body-failure tests — and 7 controls pass. Matches the PR body.
  • PR head 6c8f2784: 56/56 across the new suite plus both existing keepalive suites (45).
  • npm run typecheck clean on the head; git diff --check clean; diff is 2 files (11 lines in src/cache-keepalive.ts + a 238-line test file); no lockfile/config/kernel churn; head contains current master (no rebase needed).
  • CI (live-checked at review time via the mirror check cache): 0 failures; the main test lane and the guard lanes were still running/queued — the final green is CI's to give.

Layer judgment (root cause, not symptom). The producer of the defect is resp.text().catch(() => "") in firePing (src/cache-keepalive.ts:386 on master): the empty-string fallback was meant for non-OK responses whose body is irrelevant, but it also swallows body-read rejections on 200 responses, which then flow into the usage parser and get misread as "prefix dead". The fix moves the distinction to the decision point — a read rejection is routed to handleFailure (the same three-consecutive-failure budget + normal cadence as network/HTTP failures), while a successfully read zero/missing-usage body keeps the existing disarm. That is the mechanism that produces the defect, fixed at its source; the 429 branch and the ownership guard are preserved.

Regression review — existing behaviors on the touched path, each checked:

  1. 429 ping → immediate disarm: kept (readable and unreadable 429 bodies both disarm on the first response; tests 5–6).
  2. Non-429 non-OK ping (e.g. 500) → 3-strike budget: kept; only the log detail for an unreadable 500 body changes from http 500: to response body: <err> (log wording, not behavior; test 7).
  3. 200 with cache_read > 0 → reschedule + streak reset: unchanged (recovery half of test 1).
  4. 200 completely received with zero/missing/invalid usage → disarm prefix-dead: unchanged (tests 9–11).
  5. Rearm ownership guard (stale in-flight body vs replacement snapshot): unchanged, and now explicitly pinned (test 8 — passes on master too, confirming the guard pre-exists).
  6. Ping request bytes: unchanged (test asserts requests[0] === requests[1]).
  7. pingCount semantics: unchanged — only complete 200 responses count, consistent with network failures.

Intended behavior change (disclosed per policy): a 200 response whose body fails to read (truncated Content-Length, bad gzip) — old: single failure → immediate disarm prefix-dead(cache_read=0), no further pings; new: counted in the existing 3-consecutive-failure budget, retried at the normal cadence, disarms after 3 as errors(response body: …). That is the fix itself.

Test validity: real loopback node:http server + native Fetch (only status capture and read-counting are mocked); truncated-Content-Length and invalid-gzip bodies genuinely reject response.text(); assertions have teeth (if the read did not fail, the failStreak/pingCount/disarm-state assertions would flip); deterministic scheduler/clock seams per repo convention; ephemeral port per #360; env saved/restored. Both gate directions are covered (triggers on 200 body failure; does NOT trigger on 429 / complete miss / rearm).

Auto-merge gate (§7.4): single-module, regression-tested, clean diff, references its issue — but the failure-routing/fallback semantics change is "fallback/default-value change" territory, so it stays human-merge. No code changes needed from me; nothing to fix on the branch.

8问(结构化审核清单):

  1. 是什么问题:空闲期缓存保活 ping 收到 HTTP 200 头后读响应体失败(截断/坏 gzip)时,被空串兜底当成"缓存已死",一次失败就拆除快照、永久停止后续保活。
  2. 什么原因导致的:firePing 里 resp.text().catch(() => "")([PR] feat: idle prompt-cache keepalive for the Anthropic wire (#3376) #3390 引入)把"响应体读不出来"和"响应体为空"混为一谈——读失败被吞成空串,再被 usage 解析读成 cache_read=0,走了 prefix-dead 拆除路径。修法把判定移回源头(读失败单独路由进既有预算),不是下游过滤。
  3. 影响范围与严重程度:所有开着 Anthropic 保活的会话;触发条件 = ping 恰好遭遇传输/解压故障(偶发)。严重度 p2:保活提前静默停止 → 空闲后缓存按 TTL 自然过期、下次真请求冷重建(多花钱);下一真实请求会重新 arm,无数据丢失、无不可用。
  4. 怎么修复的:把"响应体读失败"并入既有的三连败预算 + 正常节拍重试(与网络/HTTP 失败同路径);429 仍立即拆除;完整读到但 usage 为零/缺失仍按原策略拆除;rearm 所有权守卫不变。
  5. 是否有回归问题:无。逐项核过:429 立即拆除(保持)、非 429 非 OK 走预算(保持,仅不可读 500 的日志措辞从 http 500: 变为 response body: <err>)、命中续期+清零连败(保持)、完整零/缺 usage 拆除(保持)、rearm 所有权守卫(保持,新测试钉住)、ping 请求字节(不变,测试断言两次请求体相同)、pingCount 语义(不变)。
  6. 是否破坏缓存:不。ping 请求字节不变,上游前缀缓存不受影响;只改 bili 本地保活引擎的重试策略,不涉及缓存失效/去重键。
  7. 是否改了设计方向:否。同一机制内把一类失败并入既有预算,无架构/边界/数据流转向。
  8. 是否引入了新的配置:否。无新 env/config/默认值,重试预算常量不变。

判定:可合(mergeable)——方向正确(根因级修复)、diff 最小且干净、4 个新失败在 master 上真实复现原 bug 且在 head 上通过、既有行为逐项核过无回归。等 CI 全绿(test 车道与 guard 车道提交时仍在跑),合并权在你。

Suggested labels for the PR (not applied — the source PR's labels are the author's to manage): bug, p2, scope/wiring.

@ranxianglei ranxianglei added bug Something isn't working p2 scope/other labels Nov 14, 2026
@ranxianglei
ranxianglei merged commit 645a5f7 into ranxianglei:master Nov 14, 2026
84 of 120 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working p2 scope/other

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A keepalive response-body failure is treated as a dead prompt cache

2 participants