Repository navigation
[PR] fix: draining proxy no longer claims its freed port — same-lane newcomer ladders instead of dying twin-collision (#4982) - #5016
Conversation
…n misclassification no longer kills same-lane newcomers (#4982) A proxy entering graceful shutdown releases its listening socket immediately but kept its registry marker until the drain finished (seconds-to-minutes later). During that window the marker still claimed the port while a different instance could bind it; a same-lane newcomer hitting EADDRINUSE then misclassified the draining predecessor as its twin and died with the adopt signal (exit 3), stranding its launcher behind a cross-lane stranger it is forbidden to attach to (#1225). CI hit this in the #3203 concurrent-sessions e2e (release v0.1.209, run 40574417718). Two complementary fixes: - shutdown() stamps portReleasedAt on its own marker the moment the listener closes; classifySameLanePeer skips stamped entries, so a draining instance can never be mistaken for the port holder again (predecessor-wait consumers keep counting them, preserving #3044/#1723). - the late EADDRINUSE path now verifies the claimed twin's socket ownership via /__bili/health (pid + instance identity, the #4341 criterion) before declaring twin-collision; an unverified claim falls through to the plain foreign-holder policy and ladders. tests/issue4982-draining-twin-ladder.test.ts pins both halves end to end: a real draining predecessor + silent squatter (newcomer must ladder, not die) and an unstamped marker variant (verification demotes the claim, logged as 'no longer answers as itself').
📦 Built Package ArtifactBranch: Latest build — install this exact versionnpm install -g billion-context@0.0.6594-pr.5016.b05621dThis exact version is immutable and reproducible. The rolling Recent builds (commit → version)
Download artifact
tar xzf billion-context-pr5016.tgz
npm install -g packageInstall instructions are refreshed on each push. |
📦 Merged Build PublishedThis merge to master was published as npm install -g billion-context@0.0.895-master.b08fbf3 # this exact build
npm install -g billion-context@master # always the newest merged build
|
(ai: vllm-qwen/qwen3.8-27b) 查重: 不重复 —— 本条是 PR #5016(分支 标签: 现状: PR 已合并入 master(merge commit |
(ai: vllm-qwen/qwen3.8-27b) 合并后核验进行中,先报一个发现:PR 自带的 e2e 测试 B 存在确定性失败(断言自相矛盾),master CI 会因此持续红。 已完成的核验:
下一步(无决策依赖,直接做):开一个 follow-up 小修 PR,把测试 B 的否定断言钉到真正的 twin 判决标记上(排除降级 warn 行),引用本 PR;并在 #4599(CI 总控)留一行。最终八问结论随 follow-up PR 一起给出。 |
(ai: vllm-qwen/qwen3.8-27b) 跟进修复已完成并本地验证通过,follow-up PR 正在推送(稍后给链接)。 在修断言的过程中又挖出第二处潜伏缺陷——两处都在「只有被测修复生效时才会执行」的路径上,所以合并前 CI 只炸了第一处、第二处被掩盖:
验证:该文件 6/6 连跑两轮全绿(含两个 e2e)、typecheck 干净; |
(ai: vllm-qwen/qwen3.8-27b) 收工:主修复验证正确、已合入已发布;唯一残留缺陷(它自带的回归测试构造性无法通过)已在 follow-up PR #5019 修好,待人工合并。 结构化八问(针对本工作会话):
补充:本地验证 6/6 连跑两轮(含两个 e2e)+ typecheck 干净;全量判定 = #5019 的 CI。CI 协调:已按政策在 #4599 留一行结论,master 待 #5019 合并即恢复绿。本镜像页无需再动作,等人工合并 #5019 即可。 |
Two latent defects in tests/issue4982-draining-twin-ladder.test.ts shipped with PR #5016, both on paths that only run once the fix under test works: 1. Self-contradictory assertion: the demotion warn added by the fix names "twin-collision" in its own text (src/server.ts), while test B asserts doesNotMatch(/twin-collision/) over the whole log — impossible to satisfy, so test B fails deterministically on every CI run. Pin the verdict by its unique log marker (shared by the early #4341 and late paths) instead of the substring; test A's parallel assertion gets the same marker. 2. Teardown deadlock: silentTcpListener accepted probe connections and held them; a socket with no data listener stays paused, so peer EOF is never delivered once the probing child dies, leaving CLOSE-WAIT sockets that pin squatter.close() forever (verified via ss during the hang) and the test ends by node:test cancellation. Drain each connection (sock.resume()) and destroy it on EOF. Test-only change; no production behavior touched.
…rsistence adopt-exit (option A) Ruling (owner-approved 2026-10-31): for an unverified unstamped same-lane claim the early pre-persistence gate (#4977) wins over #5016's late ladder. The second row of tests/issue4982-draining-twin-ladder.test.ts now asserts the code-3 twin-collision adopt-exit BEFORE index.db is opened instead of ladder-off; the draining (portReleasedAt-stamped) row keeps asserting the ladder. Fixture cleanup hygiene was already finally-wrapped upstream, nothing to add. Refs #5018 #5004 #4982 #5016
人话版(中文)
一个会话刚结束、另一个同 lane 会话同时启动时,新会话的代理进程有小概率被误判成"重复实例"直接处死(exit 3),启动器接管失败,整个 launch 挂掉。CI 在 release v0.1.209(PR #4982)的 #3203 并发会话 e2e 里踩中了这个毫秒级竞态(run 40574417718;master 定时跑 run 40525536885 也有过完全相同的签名)。
根因:优雅退出的代理一收到 SIGTERM 就立刻释放监听端口,但注册表里的"我占着 18787"记录要等 drain/flush 完才删(keep-alive 连接 pin 住时可长达分钟级)。窗口期内:端口已空、进程还活、记录还在 → 跨 lane 的新代理绑上这个空端口 → 同 lane 的新进程 bind 撞 EADDRINUSE,晚期路径只信注册表 + pid 存活,把正在退出的旧代理误判成自己的 twin,fail-fast 退出码 3。而真正握着端口的跨 lane 进程按 #1225 设计不可接管 → 启动器无实例可 adopt → launch 失败。
修复(两层):
portReleasedAt戳;classifySameLanePeer跳过带戳条目 —— 退出中的进程不再声称占有已放掉的端口。(前驱等待类消费者findSameLanePredecessor仍计,保住 autoRestartOnUpdate + occupied target port -> orphan process leak: new instance silently hops to the next port, old instance never exits #3044/port-zone 重启竞态:sticky 端口单调漂移,升级依附与无界自增互相矛盾(#1660 后续) #1723 的 wait-predecessor。)/__bili/health(pid + instanceId,Pi duplicate startup writes index.db before twin adoption and disables unified storage #4341 同款判据)验证被指认 twin 是否真的还握着这个 socket;验证不过按外来占用处理,ladder +1 继续服务。测试:
tests/issue4982-draining-twin-ladder.test.ts端到端钉死两半 —— 真实 draining 前驱 + 静默占位者(新进程必须 ladder 而非死),以及无戳 marker 变体(验证降级并留日志 "no longer answers as itself")。修复前这两个集成测试都会以 exit 3 复现 CI 挂法。English summary: a proxy entering graceful shutdown released its listening socket immediately but kept claiming the port in its registry marker until the drain finished (seconds–minutes later, pinned by keep-alives). A same-lane newcomer hitting EADDRINUSE in that window misclassified the draining predecessor as its twin and died with the adopt signal (exit 3); the launcher then had nothing attachable (the real holder was cross-lane, forbidden by #1225) and the launch failed. Fix: (1) stamp
portReleasedAton the marker the moment the listener closes so port-ownership classification skips draining instances; (2) verify the claimed twin's live socket ownership via health before declaring twin-collision, falling through to the ladder policy on an unverified claim. Both halves pinned by new end-to-end tests.