Skip to content

[PR] fix: draining proxy no longer claims its freed port — same-lane newcomer ladders instead of dying twin-collision (#4982) - #5016

Merged
ranxianglei merged 1 commit into
masterfrom
fix/4982-draining-twin-ladder
Oct 28, 2026
Merged

ranxianglei merged 1 commit into
masterfrom
fix/4982-draining-twin-ladder

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

人话版(中文)

一个会话刚结束、另一个同 lane 会话同时启动时,新会话的代理进程有小概率被误判成"重复实例"直接处死(exit 3),启动器接管失败,整个 launch 挂掉。CI 在 release v0.1.209(PR #4982)的 #3203 并发会话 e2e 里踩中了这个毫秒级竞态(run 40574417718;master 定时跑 run 40525536885 也有过完全相同的签名)。

根因:优雅退出的代理一收到 SIGTERM 就立刻释放监听端口,但注册表里的"我占着 18787"记录要等 drain/flush 完才删(keep-alive 连接 pin 住时可长达分钟级)。窗口期内:端口已空、进程还活、记录还在 → 跨 lane 的新代理绑上这个空端口 → 同 lane 的新进程 bind 撞 EADDRINUSE,晚期路径只信注册表 + pid 存活,把正在退出的旧代理误判成自己的 twin,fail-fast 退出码 3。而真正握着端口的跨 lane 进程按 #1225 设计不可接管 → 启动器无实例可 adopt → launch 失败。

修复(两层):

  1. 根因卫生:shutdown() 释放监听 socket 的瞬间就给本实例 marker 打 portReleasedAt 戳;classifySameLanePeer 跳过带戳条目 —— 退出中的进程不再声称占有已放掉的端口。(前驱等待类消费者 findSameLanePredecessor 仍计,保住 autoRestartOnUpdate + occupied target port -> orphan process leak: new instance silently hops to the next port, old instance never exits #3044/port-zone 重启竞态:sticky 端口单调漂移,升级依附与无界自增互相矛盾(#1660 后续) #1723 的 wait-predecessor。)
  2. 纵深防御:晚期 EADDRINUSE 路径在宣布 twin-collision 前,先用 /__bili/health(pid + instanceId,Pi duplicate startup writes index.db before twin adoption and disables unified storage #4341 同款判据)验证被指认 twin 是否真的还握着这个 socket;验证不过按外来占用处理,ladder +1 继续服务。

测试:tests/issue4982-draining-twin-ladder.test.ts 端到端钉死两半 —— 真实 draining 前驱 + 静默占位者(新进程必须 ladder 而非死),以及无戳 marker 变体(验证降级并留日志 "no longer answers as itself")。修复前这两个集成测试都会以 exit 3 复现 CI 挂法。


English summary: a proxy entering graceful shutdown released its listening socket immediately but kept claiming the port in its registry marker until the drain finished (seconds–minutes later, pinned by keep-alives). A same-lane newcomer hitting EADDRINUSE in that window misclassified the draining predecessor as its twin and died with the adopt signal (exit 3); the launcher then had nothing attachable (the real holder was cross-lane, forbidden by #1225) and the launch failed. Fix: (1) stamp portReleasedAt on the marker the moment the listener closes so port-ownership classification skips draining instances; (2) verify the claimed twin's live socket ownership via health before declaring twin-collision, falling through to the ladder policy on an unverified claim. Both halves pinned by new end-to-end tests.

…n misclassification no longer kills same-lane newcomers (#4982)

A proxy entering graceful shutdown releases its listening socket immediately but kept its registry marker until the drain finished (seconds-to-minutes later). During that window the marker still claimed the port while a different instance could bind it; a same-lane newcomer hitting EADDRINUSE then misclassified the draining predecessor as its twin and died with the adopt signal (exit 3), stranding its launcher behind a cross-lane stranger it is forbidden to attach to (#1225). CI hit this in the #3203 concurrent-sessions e2e (release v0.1.209, run 40574417718).

Two complementary fixes:
- shutdown() stamps portReleasedAt on its own marker the moment the listener closes; classifySameLanePeer skips stamped entries, so a draining instance can never be mistaken for the port holder again (predecessor-wait consumers keep counting them, preserving #3044/#1723).
- the late EADDRINUSE path now verifies the claimed twin's socket ownership via /__bili/health (pid + instance identity, the #4341 criterion) before declaring twin-collision; an unverified claim falls through to the plain foreign-holder policy and ladders.

tests/issue4982-draining-twin-ladder.test.ts pins both halves end to end: a real draining predecessor + silent squatter (newcomer must ladder, not die) and an unstamped marker variant (verification demotes the claim, logged as 'no longer answers as itself').
@ranxianglei
ranxianglei merged commit b08fbf3 into master Oct 28, 2026
50 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: fix/4982-draining-twin-ladder (350e833)

Latest build — install this exact version

npm install -g billion-context@0.0.6594-pr.5016.b05621d

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Recent builds (commit → version)

commit version build
350e833 billion-context@0.0.6594-pr.5016.b05621d build

Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr5016.tgz
npm install -g package

Install instructions are refreshed on each push.

@github-actions

Copy link
Copy Markdown
Contributor

📦 Merged Build Published

This merge to master was published as 0.0.895-master.b08fbf3 (rolling master tag).

npm install -g billion-context@0.0.895-master.b08fbf3  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

@ranxianglei ranxianglei changed the title fix: draining proxy no longer claims its freed port — same-lane newcomer ladders instead of dying twin-collision (#4982) [PR] fix: draining proxy no longer claims its freed port — same-lane newcomer ladders instead of dying twin-collision (#4982) Oct 28, 2026
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重: 不重复 —— 本条是 PR #5016(分支 fix/4982-draining-twin-ladder)的镜像页,对应原始缺陷 issue #4982;检索 open+closed 无其他同机制条目(#3044/#1723 wait-predecessor、#4341 health 判据、#1225 cross-lane 不可接管均为被引用先例,非重复)。

标签: bug scope/wiring p2(保留已有 pr) —— draining 窗口内同 lane 新进程被误判 twin 处死、launch 失败,属协议接线/实例生命周期面的竞态缺陷;仅毫秒级竞态触发(会话结束与同 lane 启动重叠),有重试绕行,定 p2。跨车道通用修复,不打客户端标签。

现状: PR 已合并入 master(merge commit b08fbf361,head 350e833),已随 master rolling build 0.0.895-master.b08fbf3 发布 —— 无需重做,进入合并后核验:通读 diff(src/instance.ts / src/port-owner.ts / src/server.ts + 新增 e2e 测试)、清点触碰路径的既有行为、本地定向跑新测试。结论稍后按八问补上。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

合并后核验进行中,先报一个发现:PR 自带的 e2e 测试 B 存在确定性失败(断言自相矛盾),master CI 会因此持续红。

已完成的核验:

下一步(无决策依赖,直接做):开一个 follow-up 小修 PR,把测试 B 的否定断言钉到真正的 twin 判决标记上(排除降级 warn 行),引用本 PR;并在 #4599(CI 总控)留一行。最终八问结论随 follow-up PR 一起给出。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

跟进修复已完成并本地验证通过,follow-up PR 正在推送(稍后给链接)。

在修断言的过程中又挖出第二处潜伏缺陷——两处都在「只有被测修复生效时才会执行」的路径上,所以合并前 CI 只炸了第一处、第二处被掩盖:

  1. 断言自相矛盾(上一条已报):降级 warn 文案自带 "twin-collision" 字面量,测试却对整份日志 doesNotMatch(/twin-collision/) → 构造性必挂。修法:否定断言改钉真正的 twin 判决唯一日志标记 already served by a same-build bili instance(早期 Pi duplicate startup writes index.db before twin adoption and disables unified storage #4341 路径与晚期路径共用),测试 A 的平行断言一并对齐。
  2. 清理死锁(新发现):silentTcpListener 占位者 accept 后持住连接;无 data 监听的 socket 处于 paused 态,探测子进程死后 peer EOF 永远不投递(ss 实测 4 个 CLOSE-WAIT 钉死 squatter.close()),测试以 node:test cancellation 收尾。修法:sock.resume() 排空 + EOF 即 destroy。

验证:该文件 6/6 连跑两轮全绿(含两个 e2e)、typecheck 干净;silentTcpListener 仅本文件使用,无外溢。纯测试改动,不碰任何生产行为。 合入前 master CI 在该文件上持续红,此 PR 合入即消除。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

收工:主修复验证正确、已合入已发布;唯一残留缺陷(它自带的回归测试构造性无法通过)已在 follow-up PR #5019 修好,待人工合并。

结构化八问(针对本工作会话):

  1. 是什么问题:PR [PR] fix: draining proxy no longer claims its freed port — same-lane newcomer ladders instead of dying twin-collision (#4982) #5016([PR] release v0.1.209 #4982 draining-twin 误判修复)本体正确并已合并发布(b08fbf3 → 0.0.895-master.b08fbf3),但它自带的回归测试 tests/issue4982-draining-twin-ladder.test.ts 构造性不可能通过——合并后每次 CI 在该文件上必红。
  2. 什么原因导致的:两处测试侧缺陷,都在「只有被测代码生效时才会执行」的路径上:① 修复新增的降级 warn 文案自带 "twin-collision" 字面量(...instead of dying twin-collision (#4982)),测试却对同一份日志同时断言 match(/no longer answers as itself/) + doesNotMatch(/twin-collision/) → 自相矛盾、必挂;② silentTcpListener 静默占位者 accept 探测连接后永久持有,无 data 监听的 socket 处于 paused 态,探测子进程死后 peer EOF 永不投递 → CLOSE-WAIT 钉死 close() 永不返回(ss 实测 4× CLOSE-WAIT 归测试进程所有),测试以 node:test cancellation 收尾。② 在①之前被掩盖(合并前 CI 绿、合并后红)。
  3. 影响范围与严重程度:CI 全 shard/全平台在该文件确定性红(全量套件跑 tests/*.test.ts);生产行为与终端用户零影响。严重度 = CI 可信度损伤——继承性红会淹没真实失败,但无用户可见缺陷。
  4. 怎么修复的:① twin-collision 判决断言改钉两条真实判决路径(早期 Pi duplicate startup writes index.db before twin adoption and disables unified storage #4341 路径 src/server.ts:653/663 + 晚期路径 :1467)共用的唯一日志标记 already served by a same-build bili instance(两条 warn 均不含该串),测试 A 的平行断言一并对齐;② fixture 加 sock.resume() 排空 + EOF 即 destroy,死 prober 不再能钉住 close()。
  5. 是否有回归问题:无。纯测试文件改动(+12/-4,单文件),生产代码零触碰;silentTcpListener 仅本文件使用;标记串经全仓核实只出现在两条真实 exit-3 判决路径;单元测试断言(classify 跳过戳条 / markPortReleased 幂等 / verifyEntryOwnsPort 预算语义)全部保持原义。既有行为清点:[PR] fix: draining proxy no longer claims its freed port — same-lane newcomer ladders instead of dying twin-collision (#4982) #5016 两层修复合义(portReleasedAt 打戳 + health 验证)、wait-predecessor(autoRestartOnUpdate + occupied target port -> orphan process leak: new instance silently hops to the next port, old instance never exits #3044/port-zone 重启竞态:sticky 端口单调漂移,升级依附与无界自增互相矛盾(#1660 后续) #1723)、跨 lane 禁接管(Launcher proxy reuse (ensureProxyRunning) breaks per-run isolation: shared-fate lifecycle + no code/client identity in compatibility check #1225)均不受影响。
  6. 是否破坏缓存:不涉及。
  7. 是否改了设计方向:不涉及——[PR] fix: draining proxy no longer claims its freed port — same-lane newcomer ladders instead of dying twin-collision (#4982) #5016 主修复架构不动,仅其自带测试的判决钉法被修正。
  8. 是否引入了新的配置:不涉及。

补充:本地验证 6/6 连跑两轮(含两个 e2e)+ typecheck 干净;全量判定 = #5019 的 CI。CI 协调:已按政策在 #4599 留一行结论,master 待 #5019 合并即恢复绿。本镜像页无需再动作,等人工合并 #5019 即可。

ranxianglei pushed a commit that referenced this pull request Oct 28, 2026
Two latent defects in tests/issue4982-draining-twin-ladder.test.ts shipped
with PR #5016, both on paths that only run once the fix under test works:

1. Self-contradictory assertion: the demotion warn added by the fix names
   "twin-collision" in its own text (src/server.ts), while test B asserts
   doesNotMatch(/twin-collision/) over the whole log — impossible to satisfy,
   so test B fails deterministically on every CI run. Pin the verdict by its
   unique log marker (shared by the early #4341 and late paths) instead of
   the substring; test A's parallel assertion gets the same marker.

2. Teardown deadlock: silentTcpListener accepted probe connections and held
   them; a socket with no data listener stays paused, so peer EOF is never
   delivered once the probing child dies, leaving CLOSE-WAIT sockets that pin
   squatter.close() forever (verified via ss during the hang) and the test
   ends by node:test cancellation. Drain each connection (sock.resume()) and
   destroy it on EOF.

Test-only change; no production behavior touched.
ranxianglei pushed a commit that referenced this pull request Oct 31, 2026
…rsistence adopt-exit (option A)

Ruling (owner-approved 2026-10-31): for an unverified unstamped same-lane
claim the early pre-persistence gate (#4977) wins over #5016's late ladder.
The second row of tests/issue4982-draining-twin-ladder.test.ts now asserts
the code-3 twin-collision adopt-exit BEFORE index.db is opened instead of
ladder-off; the draining (portReleasedAt-stamped) row keeps asserting the
ladder. Fixture cleanup hygiene was already finally-wrapped upstream, nothing
to add.

Refs #5018 #5004 #4982 #5016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant