Skip to content

[PR] fix: abort superseded cache keepalive requests - #7143

Open
Afloat16 wants to merge 1 commit into
ranxianglei:masterfrom
Afloat16:2026-11-18_cancel-superseded-keepalives
Open

Afloat16 wants to merge 1 commit into
ranxianglei:masterfrom
Afloat16:2026-11-18_cancel-superseded-keepalives

Conversation

@Afloat16

@Afloat16 Afloat16 commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #7142.

Problem

A disarmed or superseded keepalive snapshot loses its registry entry and cadence timer, but its active HTTP request remains alive until the existing 60-second deadline. A slow upstream can therefore keep a connection occupied after the snapshot is no longer useful.

The regression uses a real loopback HTTP server and an undici dispatcher limited to one connection. The old ping is held either before headers or during response-body consumption. After invalidation, the unchanged implementation prevents an ordinary request from reaching that same upstream before the test deadline.

Change

Give each in-flight snapshot its own AbortController. Replace or delete registry ownership before aborting the old request, so its rejection cannot alter the replacement's failure count or schedule. Keep the controller active through response.text() and clean up in finally.

pingInFlight now remains true through response-body consumption, so the existing five-second busy-skip guard also covers that phase and prevents another ping from starting while the previous body is still being read.

The timeout value remains 60 seconds. Its signal forwards the timeout reason to the snapshot controller through response-body consumption, with the listener removed after completion. This avoids requiring AbortSignal.any, which is not available throughout the declared Node >=20 range.

The ping request body, cache eligibility, retry budget, cadence and configuration retain their existing definitions. An ineligible same-credential forward that leaves the snapshot unchanged also leaves its active ping running.

Validation

  • Eight original regressions fail on current master 548e3960: replacement, disabling, credential rotation and concurrency eviction, each while waiting for headers and while consuming the body. Two unchanged-snapshot controls pass on that baseline.
  • The tests observe the old server response close, verify a successful ordinary HTTP request through the same single-connection dispatcher, and check that replacement snapshots can still record a hit and schedule exactly once.
  • A further real-HTTP regression removes AbortSignal.any temporarily to cover the compatibility requirement.
  • All 73 tests across cache-keepalive, keepalive-body-failures, keepalive-breakpoint-scope and the new regression file pass, with no skips.
  • npm run typecheck, npm run build, the test-import-graph guard and git diff --check pass.
  • Independent review checked cancellation ordering, body-read lifetime, stale-entry isolation and the actual connection-release assertions. The full platform/shard verdict remains with remote CI.

Prior work and scope

Checked all 258 prior fork refs, the seven actual keepalive source variants, and relevant upstream open/closed issues and PRs. #6078 isolates stale response decisions; #6739 cancels invalidated cadence timers. Neither cancels the active upstream HTTP exchange addressed here.

This change is limited to the shared host keepalive lifecycle and its regression tests. It changes no wire/message shape, persisted state, configuration, model-facing text, kernel code or compression-mode-specific path.

Primary contributor: Afloat16.

Let each snapshot own cancellation through response-body consumption and release its upstream connection when replaced or disarmed. Preserve the existing timeout and failure policy, including early Node 20 support.

Fixes ranxianglei#7142
@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

查重: 不重复 — this is the only open [PR] row for thread #7142; the only other open keepalive PR is #6650 (stream-keepalive watchdog re-arm — different subsystem, different mechanism). Cited prior work verified: #6078 (stale response-body isolation) and #6739 (invalidated cadence-timer cancellation) are both merged/closed and address adjacent gaps — neither aborts an already in-flight upstream exchange, matching the PR's claim.

标签: bug + p2 + scope/other applied to #7142 and this PR row (host-side keepalive lifecycle defect; bounded connection occupation with a 60 s workaround, not a usability or cache-leak class).

Review of PR #7143 (head 7996c494, base = current master 548e3960, up to date):

① Direction — the root cause in #7142 is real and correctly located: disarm() and snapshot replacement cancel the cadence timer and remove the registry entry, but the in-flight fetch carries only AbortSignal.timeout(60 s) — nothing ties the request's lifetime to the snapshot's. The fix gives each in-flight ping an entry-owned AbortController and aborts it on every ownership change (replace / disable / credential-rotate / concurrency-evict), while waiting for headers or consuming the body. That is a fix at the producer (request lifetime), not a mask.

② Regression review — existing behaviors on the touched path, itemized:

  • arm/tier/cadence/config resolution, ping body construction (max_tokens=1, stream=false), failure budget (3-strike), 429 → immediate disarm, prefix-dead disarm, stale-entry guards, status surface, log lines: unchanged (the firePing body is only re-indented under the new try/finally).
  • Intentional behavior changes (all beneficial, disclosed here):
    1. in-flight pings are now aborted on disarm/replace/evict/rotate — the fix itself;
    2. pingInFlight is now held through response.text() (previously cleared right after headers) → the 5 s busy-skip window now covers body consumption; side effect: a second concurrent ping can no longer be started mid-body (duplicate-ping race eliminated);
    3. the 60 s deadline now covers the whole exchange including body reads — on master the post-headers body read was unbounded (a stalled body could hang the ping continuation until TCP death). Note: the PR body's "preserves the 60-second deadline" is slightly imprecise — the deadline value is unchanged, its coverage is extended.
  • No wire/message-shape, persistence, config, kernel, or model-facing-text change.

③ Code — verified by reading the full file at both heads:

  • registry ownership is replaced/removed before the old request is aborted, so the stale rejection hits the handleFailure staleness check and cannot spend the replacement's failure budget or schedule (pinned by the pre-existing "a failed old response body cannot spend the replacement snapshot's budget" test plus the new tests asserting failStreak 0 and exactly one pending timer on the replacement);
  • no path exists where an abort-induced rejection is counted against the current entry — only the timeout path counts, same as master;
  • the timeout relay via addEventListener (instead of AbortSignal.any) is justified: engines.node is >=20 and AbortSignal.any needs 20.3+;
  • controller cleared in finally, timeout listener removed — no leak.

④ Test validity — real http.Server on listen(0) + real undici single-connection Agent + real fetch abort semantics (not mocks); the connection-release assertion has teeth: an ordinary request through the same 1-connection dispatcher must succeed within 1 s, which times out on master. The unmodified-snapshot controls pin the negative direction (an ineligible same-credential forward must NOT abort the live ping). Verified locally:

  • PR head: 73/73 pass, 0 skips across cache-keepalive, keepalive-body-failures, keepalive-breakpoint-scope, issue7142-keepalive-flight; npm run typecheck, npm run build, check:test-import-graph, git diff --check all clean;
  • master's cache-keepalive.ts + the new test file: 9 fail (the 8 regressions plus the AbortSignal.any test, which fails on master for the same root reason) and 2 controls pass — exactly as the PR claims.

⑤/⑤b Diff & code cleanliness — 2 files (+266/−42), every line on-topic, no whitespace churn; naming/comments/constants consistent with the file; no new dependencies (undici is already a devDependency); test file follows the issueNNNN-*.test.ts convention.

CI (mirror cache, 10:59:59Z snapshot): 14 success / 11 skipped / 5 in progress / 0 failed.

Verdict: no blocking issues — mergeable once the remaining CI shards are green. The two side-effect behavior changes (②.2/②.3) deserve a line in the PR body for future readers; the fix itself is minimal, correctly layered, and properly pinned.


结构化八问(供人类审阅):

  1. 是什么问题:keepalive 快照被解除/替换/淘汰后,它已在途的 ping 请求不会被取消,继续占用上游连接直到 60 秒超时,期间同一 dispatcher 上的正常请求被阻塞。
  2. 什么原因导致的:ping 的 fetch 只带 60 秒超时信号,没有任何与快照生命周期绑定的中止通道——disarm/替换只取消了定时器和注册表条目,不取消在途请求(根因是请求生命周期未绑定快照生命周期,不是 stale-entry 检查的问题)。
  3. 影响范围与严重程度:仅影响 keepalive 功能(Anthropic wire + 显式启用);上游慢 + 快照失效时单条连接被占用最长 60 秒,同 dispatcher 正常请求延迟;有绕行(等 60 秒超时)→ p2。
  4. 怎么修复的:每个在途 ping 持有一个 entry 拥有的 AbortController;所有所有权变更(替换/禁用/凭证轮换/并发淘汰)先移除/替换注册表条目、再 abort 旧请求;旧请求的 rejection 被既有 stale-entry 检查隔离,不能消耗新快照的失败预算或调度;60 秒超时通过事件监听器中继到快照 controller(不用 AbortSignal.any,兼容 Node 20.0–20.2)。
  5. 是否有回归问题:未发现回归。逐项核对既有行为:arm/tier/周期/配置解析、ping 体、失败预算、429 处理、prefix-dead disarm、stale-entry 隔离、状态面、日志行——均不变。两个有意的行为变更(已披露):① pingInFlight 现在保持到 body 读完 → 5 秒 busy-skip 窗口延伸到 body 读取(副作用:消除重复并发 ping 竞态);② 60 秒 deadline 现在覆盖 body 读取(master 上 headers 之后的 body 读取无超时,stalled body 可无限挂起)。两者均为改善。
  6. 是否破坏缓存:否。keepalive 本身是 Anthropic 前缀缓存的 TTL 刷新机制,本变更只缩短失效 ping 的存活时间;hit/miss 判定与 reschedule 逻辑不变(有测试钉住)。
  7. 是否改了设计方向:否。不改架构/边界/数据流,只在既有 keepalive 状态机内补全在途请求的生命周期。
  8. 是否引入了新的配置:否。

结论:review 通过,无阻塞问题;CI 全绿后 owner 可合并。

@ranxianglei ranxianglei added bug Something isn't working p2 scope/other labels Nov 18, 2026
@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

CI final state (live-verified, ~11:25 UTC) — my earlier CI line was from an in-progress mirror snapshot; it has since resolved:

  • All 120 check runs on head 7996c494 are completed: 57 success / 27 skipped / 0 failed. The 36 "cancelled" runs are intermediate re-runs of the same check names (a re-run wave at 11:19 Z, right after the PR was labeled); for every check name the latest run is success or skipped — no check is left cancelled or pending.
  • Full test matrix green: ubuntu × node 22/24 × shards 1–3, windows × node 22 × shards 1–5, plus all guard lanes (kernel-guard, pi-subagents-guard, config-surface-guard, version-guard, lock-sync, test-import-graph, unused-exports, docs-sync, …).
  • PR mergeable_state is currently blocked on branch protection (review), not on checks — merge is the owner's call.

No changes to my verdict: no blocking issues; CI fully green.

@Afloat16

Copy link
Copy Markdown
Contributor Author

(ai: OpenAI/GPT-6 Astra Pro)

The requested body clarification is now in place: pingInFlight remains set through response-body consumption, and the timeout value remains 60 seconds.

I found a separate required-check blocker on the unchanged heads of #7038, #7133, #7138, #7141, #7143 and #7147. For #7143, the 12:21 plan job records LABEL_EVENT: root-cause and plan scope: skip; its test matrix is skipped and the merge box shows 14 required checks as Expected. The same label-event outcome is present on #7133, #7138, #7141 and #7147.

There is also a matrix/required-context mismatch: the normal core matrix supplies 11 test names, while master requires 14, including Windows Node 24 shards 1–3. Those three are absent from the core run even when all 11 pass, as on #7038. The existing ci:full path runs 16 test legs and covers all 14 required names.

Could a maintainer apply the existing ci:full label to these six PRs so the required jobs actually run? Re-running the same root-cause event would skip again, and a core rerun would still omit the three Windows Node 24 requirements. No workflow or branch-protection changes are needed or included here.

Primary contributor: Afloat16.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Invalidated cache keepalive pings keep occupying upstream connections

2 participants