Skip to content

[PR] fix: ignore stale keepalive response bodies after rearming - #6078

Closed
Afloat16 wants to merge 1 commit into
ranxianglei:masterfrom
Afloat16:2026-11-10_keepalive-snapshot-race
Closed

Afloat16 wants to merge 1 commit into
ranxianglei:masterfrom
Afloat16:2026-11-10_keepalive-snapshot-race

Conversation

@Afloat16

Copy link
Copy Markdown
Contributor

Problem

A real request can replace a session's keepalive snapshot while an older ping awaits its response body. The existing ownership check ran before response.text(), so a stale HTTP 429 or cache miss could subsequently disarm the new snapshot and cancel its timer.

Changes

Recheck that the registry still owns the same entry immediately after body consumption. Superseded replies cannot apply stop/success decisions to a newer snapshot. Current-entry hit, miss, error and scheduling behavior remains unchanged.

Regression evidence

At base 541cf26, deterministic deferred-body cases reproduce both failures: an old 429 and an old cache miss delete the replacement 1-hour snapshot. A stale cache-hit control passes.

  • Before: tests/cache-keepalive.test.ts — 37 passed, 2 failed (the two new ownership regressions).
  • After: the same suite — 39 passed, 0 failed.
  • npm run typecheck — passed.
  • npm run build — passed.

The fork commit has the identical Git tree (8c085ce01475d6bf80eff1580a73889f3be95c21) as the locally verified commit d327946f61cd6f8323526f880d979316144895ef. The full remote CI verdict remains separate from these targeted local checks.

Prior work

Screened all 3,221 existing upstream PRs, including every Afloat16 PR (#6046–#6055). #3390 introduced keepalive; #5669/#5672 integrated through #5847 fix default TTL detection and timer liveness. This change addresses the separate body-completion ownership race.

Closes #6077

@ranxianglei ranxianglei changed the title fix: ignore stale keepalive response bodies after rearming [PR] fix: ignore stale keepalive response bodies after rearming Nov 10, 2026
@Afloat16

Copy link
Copy Markdown
Contributor Author

Could a maintainer rerun the failed Windows Node 22 shard 3 on 0d2ee1d?

The failure is the duplicate-start log assertion in issue4341-early-twin-adoption.test.ts, followed by three issue4601-ws-failure-frame.test.ts cases cancelled with pending promises. Those same cases passed in the Windows Node 22 control job on #6084. Both jobs checked out merges against 541cf26; the failing test files and the CLI/instance source files are byte-identical between those tested trees.

This PR only adds the post-body ownership recheck in firePing and its focused tests. A rerun would help establish whether the Windows failures reproduce before changing the startup or WebSocket paths.

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Duplicate check: no duplicate — searched open+closed issues/PRs for these test files and the flake signature; nothing tracks them yet.

Labels: bug, scope/ci, p3 applied (CI-only timing noise on one shard; no user-facing impact).

I verified the claim independently before acting, rather than rerunning on request alone:

  1. Run scope: on 0d2ee1d6, 25 of 26 jobs are green — only test (windows-latest, 22, 3) failed (all other Windows shards 1/2/4/5 and all Ubuntu shards passed). Shard summary from the log: # tests 2030 / # pass 2012 / # fail 1 / # cancelled 3.
  2. Failure fix: keep system prompt byte-stable for prefix cache #1 is not ok 894 — real CLI: duplicate-start burst keeps the serving process in unified mode (tests/issue4341-early-twin-adoption.test.ts:327): the assertion expected the losing twin to log exiting before persistence initialization, but the captured output only contains log-retention lines from both processes within the observation window — a startup-race timing miss under runner load, not an assertion about keepalive behavior.
  3. Failures fix: address all code review findings (condense, routing, session-id, dead code) #2–4 are the three issue4601-ws-failure-frame.test.ts cases cancelled with failureType: 'cancelledByParent' and Promise resolution is still pending but the event loop has already resolved — the file's event loop drained while subtest promises were still pending, the classic load-induced hang pattern.
  4. Diff relevance: this PR is +32 lines in two files — the one-line post-body ownership recheck in firePing (src/cache-keepalive.ts) plus its focused tests (tests/cache-keepalive.test.ts). It touches neither the CLI startup path nor the WS frame paths, and cache-keepalive.test.ts isn't even in shard 3's file set (it ran green elsewhere in the run).
  5. Control job confirmed: I checked run 42752181460 job 129619525617 directly — completed / success, matching your report.

Action taken: reran the failed jobs on run 42749373752 (rerun accepted, run now queued) — my own judgment based on the evidence above that this is a single-shard timing flake unrelated to the diff, so no startup/WebSocket changes are warranted before seeing the rerun. If shard 3 reproduces the same failures on the rerun, that becomes a genuine signal and the two tests' timing robustness needs its own investigation (which would be filed separately, per the one-issue-per-mechanism rule).

Merge remains human-only as always.


结构化八问(中文):

  1. 是什么问题:PR [PR] fix: ignore stale keepalive response bodies after rearming #6078 的 CI 中仅 Windows Node 22 shard 3 一个 job 红——真实 CLI 双启动竞态测试的时序断言失败 + 三个 WS 测试被取消。
  2. 什么原因导致的:runner 负载下的时序抖动(flake):竞态输方进程未在观察窗口内打出预期退出日志;WS 测试文件事件循环先于 promise 结束触发 cancelledByParent。与 PR 改动无关(diff 仅 firePing 一行所有权复查 + 其测试,不触碰启动/WS 路径)。
  3. 影响范围与严重程度:仅该 PR 的 CI 门禁被卡,无用户面影响;属边缘场景噪音(p3)。
  4. 怎么修复的:未改代码。独立核实(全 run 25/26 绿、失败签名、对照组 job 实测 success、diff 相关性)后重跑失败 job;若复现则另行立项分析这两个测试的时序健壮性。
  5. 是否有回归问题:不涉及(无代码变更);已核对其余 25 个 job 全绿。
  6. 是否破坏缓存:不涉及。
  7. 是否改了设计方向:不涉及。
  8. 是否引入了新的配置:不涉及。

结论:重跑已触发,等结果即可;暂无需改动启动或 WebSocket 路径。

@ranxianglei ranxianglei added bug Something isn't working p3 scope/ci labels Nov 10, 2026
@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

Outcome: rerun is green. test (windows-latest, 22, 3) on 0d2ee1d6 completed success after the rerun triggered above; every job in run 42749373752 is now passing, so the PR's CI is fully green.

This confirms the flake diagnosis from my analysis in the previous comment: the original failure was runner-load timing noise in issue4341-early-twin-adoption.test.ts (twin-start log assertion) plus three cancelledByParent cases in issue4601-ws-failure-frame.test.ts — none of which relate to this PR's one-line firePing ownership recheck. No changes to startup or WebSocket paths were needed or made.

Remaining: owner review and merge (human-only). If those two tests start failing repeatedly on unrelated PRs, their timing robustness deserves its own issue — nothing indicates that yet.

@ranxianglei

Copy link
Copy Markdown
Owner

Landed via batch PR #6439 (merged) — closing as superseded.

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

Acknowledged — the keepalive fix landed via batch PR #6439, so this PR is superseded. Thread recap for the record: the only blocker was the Windows Node 22 shard-3 flake (twin-start timing assertion + three cancelledByParent WS cancels), which the rerun confirmed as runner-load noise — the same commit went fully green, and no startup/WebSocket changes were ever needed. Nothing outstanding on this thread; closing is yours to do.

@ranxianglei ranxianglei reopened this Nov 14, 2026
@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

Status report after the session restart — nothing was lost; all prior work is intact in the thread. Live-verified current state:

Nothing outstanding on this thread from my side: no code changes were ever needed beyond the original one-line firePing fix, and that fix is already in via #6439. No further action required.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working p3 scope/ci

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A stale keepalive response can disarm a replacement request snapshot

2 participants