Repository navigation
fix: verify sole port ownership after bind; guard proxy-origin against clobbering a live holder (#2941) - #2947
Conversation
📦 Built Package ArtifactBranch: Option A — Install this exact build (recommended)npm install -g billion-context@0.0.3347-pr.2947.0efee09This exact version is immutable and reproducible. The rolling Option B — Download artifact
tar xzf billion-context-pr2947.tgz
npm install -g packageInstall instructions are refreshed on each push. |
[bot] 🏷 审查结论(PR #2947 / #2941) 审查过程:拉取分支 ① 是什么问题:Windows 上同 lane 两个 proxy 可同时绑定同一 zone port(libuv v1.x 在 Windows TCP bind 不设 SO_REUSEADDR/SO_EXCLUSIVEADDRUSE,Node 也未暴露配置项),双实例同端口服务;且 bind 完成到实例注册之间 ② 什么原因导致的:四层——(a) 平台 socket 语义容忍对活动监听端口的并发 bind;(b) health 路由无状态,bind 后立即应答,而 pointer/registry 写入在 bind 回调之后,留下 ~200–400ms「健康但不可发现」窗口;(c) 指针文件写入无 owner 身份检查;(d) 失败的 ③ 影响范围与严重程度:影响 Windows 上 launcher spawn 的 lane proxy(zone port 场景);严重情形为双实例同端口服务、状态机分叉、指针指向错误实例、MCP children 卡 unhealthy、进程 T+~90s 静默死亡(#1821 报告)。标准 Linux/macOS 上 libuv 强制 bind 排他,新路径是纯防御第二层,不可达。 ④ 怎么修复的(四个机制,均保守设计——只凭阳性证据 yield/refuse,歧义一律放行):
⑤ 是否有回归问题:未发现。逐项审计:
⑥ 是否破坏缓存:不涉及。持久化格式/版本未变,缓存/重试/增量游标/去重键均不受影响。 ⑦ 是否改了设计方向:无架构转向。新增 ⑧ 是否引入新的配置:无。无新增 env/config/CLI flag;复用既有 BILI_LAUNCH_TOKEN/BILI_STRICT_PORT 语义。 CI 状态与拦截项处置:原 head 合并判定:代码审查通过,无回归、无遗留问题;待剩余 5 个 CI leg 转绿即可合并(结果出来我会跟进确认)。 |
[bot] 🏷 跟进:CI 状态异常说明
判断:更可能是某个 live 测试在 runner 上挂住或 runner VM 本身不健康(仓库有 CI flake 前科,如 #1896),而非 diff 引入的回归。运行中 job 的日志 blob 尚不可下载(GitHub 对 in_progress job 不发布日志),无法进一步定位到具体测试。 处置:继续监控;若总时长超过约 2 小时仍无进展,将触发该 run 重跑(stuck leg 重跑是标准运维操作,会保留其余已绿结果之外的完整重验)。有结果立即回帖。审查结论本身不受影响(见上一条八问)。 |
[bot] 🏷 纠正我上一轮(第 6 楼)的判断:CI 四条 test leg 卡死不是 runner 抖动,是 PR 自身引入的确定性回归。已定位、已修、已推两个标注 commit( 一、证据链(旧 run 日志取证 + 本地复现)旧 run(@
本地在 二、根因(三层因果)
三、修复(两个独立标注 commit,压在
|
|
ci 还是失败 |
[bot] 🏷 是的,但这一轮是进展:上一轮的挂起已修好——4 条 test leg 从「挂 90 分钟被取消」变成「~15 分钟正常跑完」,现在每条 leg 只剩一个失败用例,且已定位到根因,正在修:
|
[bot] 🏷 进展更新 —— 上一轮"ci 还是失败"的根因已修掉,现在只剩一个 leg 在抖: ① 上一轮的单一失败(probe keep-alive 残留污染 #1529 契约测试)已修复 —— ② 新 CI(run @75eab3ef5)结果:ubuntu-22 ✅ / ubuntu-24 ✅ / windows-24 ✅ / windows-22 ❌ —— 只剩这一个 leg 红,其余 16 个 job 全绿。 ③ windows-22 的两处失败都查了,证据指向负载抖动而非本 PR 回归:
两处失败模式都是超时/饿死型,且本地在 CI 实际测试的那棵树上全部确定性通过;windows leg 基线本来就比 ubuntu 慢 ~40%,c=4 并发下更容易踩到这些时间敏感测试。 ④ 已触发该 leg 重跑(rerun-failed-jobs,201 确认)。转绿后我按八问体例给最终收口回复;若再红,我会下载日志深挖 windows 特异性再继续。 |
…t clobbering a live holder (#2941) - post-bind self-probe: where socket semantics tolerate a concurrent bind onto an actively-listening port, the late binder yields to the EADDRINUSE ladder (or dies under strict-port) instead of co-serving one port from two state machines - health endpoint reports 503 not-ready until startup registration completes: closes the bind->registration window in which a concurrent same-lane launcher saw 'healthy' for an undiscoverable instance and spawned a duplicate - proxy-origin pointer write refused while a different live instance health-verifies as serving the same address (registry entry still written; draining-predecessor takeover #1723 unaffected) - attemptListen clears stale once('listening') callbacks so a successful bind after failed attempts announces exactly once - four real-server test files poll for health readiness after 'listening' to match the new contract
The four type declarations (OwnerProbeInfo/ProbeDeps/PortClaimOutcome/ PointerGuardResult) are referenced only inside port-owner.ts itself; the gate requires them to stay module-private.
75eab3e to
1bd3ad7
Compare
[bot] 🏷 重要发现 + 处置:分支在我重跑 leg 期间(01:04Z)被重写——rebase 到最新 master 并整理成 3 个干净 commit( 复现证据(新 head
也就是说当时正在跑的主 CI 必然再红(而且会再挂一轮)。 处置:把三个已审查过的修复以标注 commit 形式 cherry-pick 回新 head(全部干净应用,无冲突),已推送
本地验证(新树上定向):banner-identity 1/1、issue1452 10/10、port-owner 24/24、unused-exports gate 干净、typecheck 干净。旧的那轮注定失败的 CI 会被并发组自动取消,新一轮全量 CI 已在 说明:如果这次分支重写是有意的(比如想换方案),这三个 commit 都有独立标注、diff 很小,评审后可以直接 drop——但没有它们(或等价替代)CI 会确定性红。转绿后我按八问体例给最终收口回复。 |
[bot] 🏷 最终收口(八问体例) — CI 已全绿: 1. 是什么问题:Windows 上(luv 不做 TCP 排他绑定的任何平台)同一 zone 端口的两条 lane proxy 会双绑定成功,实例指针与 socket 持有者分裂(#2941/#1821):重复 spawn、~90s 后静默死亡、MCP 子进程卡 unhealthy。审查中又挖出三个本 PR 自身引入的次生缺陷(banner 时序竞态挂死、 2. 什么原因导致的(四个机制层):①bind→注册窗口——health 在 bind 后即回 200,而指针/注册文件晚 200-400ms 才写,并发同 lane 启动器看到「健康但不可发现」即重复 spawn;②proxy-origin 指针裸 last-writer-wins,无身份校验;③listen 失败时 3. 影响范围与严重程度:Windows lane proxy 用户面——完全可用但状态分裂,表现为静默死亡与子进程 unhealthy(#1821 实锤报告);stock 平台(bind 天然排他)仅防御性覆盖。次生缺陷影响面:banner 竞态使 CI test leg 确定性挂死(~90min 直到并发取消)、 4. 怎么修复的(机制层面):①bind 后 sole-ownership 自探测——发现异己活持有者则 ladder(+1,17 次后 ephemeral)或 die;探测到自家预注册的 5. 是否有回归问题:逐项清点过,未发现破坏。高危模式逐条核:错误处理未从 fail-fast 变吞错(strict-port/manual 路径原样 exit 1);EADDRINUSE ladder/predecessor-wait 路径未动;registry 记录写入仍无条件;阈值/时序漂移仅有有意变更(见下);输出格式变化仅 health 503 窗口(所有消费方审计兼容);两压缩模式 wire 均未触碰(改动限于 boot/wiring + admin 路由);kernel 未动;unused-exports gate、typecheck 全干净;CI 全矩阵绿(含真实客户端 e2e 套件)。 6. 是否破坏缓存:不涉及——未触碰任何缓存/重试/增量游标/去重键逻辑。 7. 是否改了设计方向:不转向。issue 清单第 2 项(starter 预算耗尽后自 spawn vs 拒绝——产品决策)与第 4 项(等 reporter 侧数据)有意搁置并已在此线程标记给 owner,其余按原方案落地。 8. 是否引入新配置:无。零新增 env/config/CLI 表面。 行为变更披露(旧值→新值):①health:bind 后至注册前( 过程备注:初版 head 的 CI 失败均定位为本 PR 引入的确定性缺陷(非 runner flake),每处修复都有本地 A/B 证据,详见第 7/13 楼;期间分支被重写(rebase+整理成 3 commit)导致两处修复丢失,我在新 head 上复现确认后以标注 commit 重新应用(cherry-pick 干净无冲突),见上一楼说明。 合并判定:可合。 CI 全绿、缺陷闭环、回归清点完毕,等 owner 合并即可。 |
📦 Merged Build PublishedThis merge to master was published as npm install -g billion-context@0.0.378-master.097e76c # this exact build
npm install -g billion-context@master # always the newest merged build
|
… x #2947) Cross-branch gap found while rebasing #3044 onto master: #2947's post-bind port-ownership check (src/port-owner.ts resolvePortClaim) gives any launch-token child the right to move +1 off a contested just-bound port. A self-restart replacement inherits its parent's BILI_LAUNCH_TOKEN, so on socket semantics that tolerate a concurrent bind onto an actively-listening port it could still hop P+1 after a 'successful' bind — the exact orphan shape #3044 kills, via the second hop path. The EADDRINUSE handler was already closed by this branch; this closes the claim path too: postBindCanLadder() in the same pure policy module, pinned in tests/listen-policy.test.ts.
… x #2947) Cross-branch gap found while rebasing #3044 onto master: #2947's post-bind port-ownership check (src/port-owner.ts resolvePortClaim) gives any launch-token child the right to move +1 off a contested just-bound port. A self-restart replacement inherits its parent's BILI_LAUNCH_TOKEN, so on socket semantics that tolerate a concurrent bind onto an actively-listening port it could still hop P+1 after a 'successful' bind — the exact orphan shape #3044 kills, via the second hop path. The EADDRINUSE handler was already closed by this branch; this closes the claim path too: postBindCanLadder() in the same pure policy module, pinned in tests/listen-policy.test.ts.
Closes #2941
Checklist mapping (one mechanism per sub-item)
[x] Item 1 — win32 reuse-disable + real-machine verification: not implementable as literally written, for two verified reasons:
src/win/tcp.c uv__tcp_try_binddeliberately does NOT set SO_REUSEADDR on Windows TCP sockets (comment cites libuv issue fix(ccr): plugin-mode sessions execute the base ccr block exclusively #1360: it "doesn't just allow binding to addresses in TIME_WAIT, it effectively allows 'stealing' a port which is in use by another application"). It also does NOT use SO_EXCLUSIVEADDRUSE (too strict for TIME_WAIT). So on stock Node/libuv/Windows a second ACTIVE listener fails with EADDRINUSE and the existing ladder works — the issue's double-bind premise does not hold for stock builds.net.Serverexposes noreuseAddr/exclusiveAddressUseoption at all.The issue's own fallback clause applies, so this PR lands the spawn-side hardening instead: a post-bind self-probe (
src/port-owner.ts). After a successful bind, the child probes its own origin's/__bili/healthup to 3× (100ms apart); if a CONCRETE foreign pid answers (round-robin between co-listeners), it yields exactly like the EADDRINUSE path — ladder +1 (ephemeral after 17 attempts) or explicit die under strict-port/manual launch. Platform-independent: neutralizes same-port co-serving even on a build/runtime that tolerates concurrent binds, without changing the availability policy. No-pid responders are never treated as foreign (conservative; pre-#1753 builds). The probe self-attributes its own pre-registration503 {ok:false, reason:"starting"}answer (see readiness contract below), so a normal boot costs ~1 loopback RTT instead of burning the full budget.[x] Item 3 — pointer identity check:
mayClobberPointergates theproxy-originwrite inannounceListening: refuse to overwrite ONLY when the existing record is a full instance file with a DIFFERENT instanceId, same URL host, recorded pid alive, AND that origin's health reports pid == recorded pid. Allows: missing/legacy records, cross-address holders (last-writer-wins by design, #1232), dead pids, draining predecessors (health down → keeps #1723 takeover), stale records (responder pid mismatch → overwrite repairs the divergence). The registry entry is still written unconditionally (it is authoritative for lane-aware discovery).[ ] Item 2 — fall-through policy flip (self-spawn vs reject): product decision, deliberately NOT changed here. Current behavior (starter budget exhausted / state dir unwritable → spawn anyway) is preserved; flagging for owner decision in the issue thread.
[ ] Item 4 — T+90s clean-death chase: blocked on reporter data (
%TEMP%\\bili-proxy-18787.logtail, bili.log window, ZCode restart events) — requested in the issue thread. Note: the silent death is consistent with the designed parent-gone exit (info line, no error line) ~7s after the last owner dies.Root cause found while implementing (the regression this PR also fixes)
A/B debugging (master 7/7 vs first draft 5/7 on
tests/port-zone-live.test.ts) exposed that making the announce path async opened a bind→registration window: the child binds, serves stateless/__bili/healthimmediately, but writes its pointer/registry entry only AFTER the new post-bind check completes (~200–400ms). A launcher returning from its wait loop — or a second concurrent same-lane launcher — could see "healthy" for an instance discovery cannot find yet, and spawn a duplicate (which then ladders off the port). Fix:/__bili/healthreturns 503{ok:false, reason:"starting"}until registration completes (flag flipped as the LAST statement ofannounceListening, so degraded boots still become ready). Every health consumer keys on HTTP ok, so one gate covers all of them: the starter's own wait loop now returns only after the instance is discoverable, and a late arrival during the window finds the starter's marker still held and converges via the existing #707 wait instead of spawning.Second defect fixed in passing: each failed
listen()left itsonce('listening')callback stacked on the server (a failed listen never fires the event), so a later successful bind fired ALL accumulated callbacks at once — double banner / double registration /MaxListenersExceededWarning: 11 listening listenerson long predecessor waits.attemptListennow clears prior listeners before re-arming; only our own callbacks ever use that event.A third timing regression was caught by local repro while chasing CI: during the post-bind gate,
/__bili/healthanswered 503{ok:false, reason:"starting"}, which the probe read as unattributable — so EVERY boot burned its full probe budget (~300ms+) probing itself before claiming the port, delaying pointer/registry publication past callers that expect registration right after bind (tests/multi-instance-live.test.tsasserts the #394 late-starter warn within ~130ms of bind and failed locally). Fixed in 1bd3ad7: the probe parses the non-ok reason body and treatsreason:"starting"as self-attribution (binds are exclusive on every shipped platform — Linux EADDRINUSE / Windows libuv #1360 — so no foreign listener can be in that state; a hypothetical co-starter would be unregistered anyway). Typical boot gate cost is now ~1 loopback RTT.Behavior changes (old → new, all deliberate)
/__bili/healthgains a transient pre-boot state: 503{ok:false, reason:"starting"}from bind until registration completes (typically ~1 loopback RTT per boot; worst case bounded by the probe budget). Old: 200 ok:true as soon as the socket accepted. Consumers: launcher spawn-wait/attach (now correctly gated — intended fix), host-native respawn checks (see 503 → their ensureProxyRunning converges via the bug: cross-process startup-window race — two concurrent launches each spawn a proxy (double writers over one sessions dir) #707 marker to attach, not double-spawn), Docker-style health scripts (transient blip per boot). Watchdog field semantics unchanged once ready.proxy-originclobber protection: a second live instance on the SAME address no longer overwrites the pointer (warn line[pointer] keeping proxy-origin held by live instance … (#2941)); readers keep pointing at the first owner. Cross-address last-writer-wins unchanged (lane-aware instance discovery — single proxy-origin file breaks same-client reattach under concurrent multi-client use #1232); draining-predecessor takeover (port-zone 重启竞态:sticky 端口单调漂移,升级依附与无界自增互相矛盾(#1660 后续) #1723) unchanged (health-down holders are overwritable).admin-probe,watchdog-health,watchdog-health-daemon,blind-tunnel-stats) now poll for health readiness afterlistening, matching the new contract. Assertions otherwise untouched.No config surface added. Kernel untouched. Both compression modes unaffected (boot/wiring + admin route only; no wire/message-shape change).
Verification
ACP_TEST_CLAUDE_NATIVE=1 tests/port-zone-live.test.ts): master 7/7; pre-fix draft 5/7 (all failures = duplicate spawn landing 18788/18789 instead of attaching); post-fix 7/7, no duplicate banners, no MaxListeners warnings.port-owner(24 tests incl. real-socket round-robin detection, starting self-attribution, end-to-end pointer guard through real state files),multi-instance-live(the timing regression caught while chasing CI, fixed above),admin-probe,watchdog-health(-daemon),blind-tunnel-stats,port-race,instance-governance,issue2187-launcher-slow-start,auto-restart,graceful-close, plus the gated live lanesport-zone-live7/7 andclaude-native55/55.npm run typecheck+npm run buildclean.