Skip to content

fix: self-restart replacement binds-or-dies on the freed port instead of hopping (#3044) - #3055

Merged
ranxianglei merged 6 commits into
masterfrom
2026-10-13_fix-3044-replacement-port-hop
Oct 14, 2026
Merged

ranxianglei merged 6 commits into
masterfrom
2026-10-13_fix-3044-replacement-port-hop

Conversation

@ranxianglei

@ranxianglei ranxianglei commented Oct 13, 2026 •

Copy link
Copy Markdown
Owner

Self-restart handover over an occupied port: bind-or-die, never hop (#3044)

Closes #3044

What was wrong

With autoRestartOnUpdate, when the old instance still held its port at spawn time (or a racing process took it during the drain window), the replacement inherited the launcher's EADDRINUSE ladder (BILI_LAUNCH_TOKEN set, not strict-port) and silently drifted to P+1, P+2, …: idle processes serving no client while the prior instance never exited. The issue reports 14 such orphans over 9 h (~8 GB private memory combined), stale never converging, and the host never learning the upgrade didn't land.

A 2026-10-13 recurrence on a clean host restart (v0.1.197, predating this fix) confirmed the leak repeats on every plugin-channel update and exposed a second trigger path in the same failure domain: a host respawn (dsh plugin channel) raced its own attach scan against a LIVE same-build instance — cmdline --port 18812 landed LISTENing on 18813 while the 18812 instance kept serving, then became the orphan. Launcher-spawned children carry no replacement marker AND the predecessor wait only matches DIFFERENT-build holders, so a same-build twin sent them straight down the ladder. Observed side effects of the coexistence: the sticky lane-port cursor (port-zone.json) ratcheted permanently with each hop, and two live instances contended on the shared state dir (persist stale-snapshot rejections).

Four sub-defects in one mechanism family (bind/handover over an occupied port):

  1. The hop itself — nothing marked the self-restart replacement as "must own exactly this port", so it got full ladder rights.
  2. Blind parent — readiness was a raw TCP probe to P: it could not distinguish "our child on P" from "a racing peer grabbed P" (parent could even exit(0) believing success while the real child idled drifted) from "child failed".
  3. Unverified kill + resume drift — child.kill() results were never verified (the surviving orphan is the reported leak), and the parent's resume-after-failure went through the same ladder, so an OLD-code copy could drift onto P+1 as its own zombie.
  4. Twin collision — an unmarked lane'd spawn racing a live SAME-BUILD holder on the same port hopped past it (the 2026-10-13 recurrence); plus a post-bind variant where even a marked replacement could move +1 off a just-bound port under concurrent-bind socket semantics (Windows: concurrent lane proxies can double-bind the same zone port (SO_REUSEADDR semantics) — instance pointer diverges from the socket owner #2941/bug: cross-process startup-window race — two concurrent launches each spawn a proxy (double writers over one sessions dir) #707/libuv#1360).

The fix

Behavior changes (old → new)

# Old New Why
a Replacement silently hops to P+1/P+2… and idles (orphan) Fails fast, exit code 1, loud log line The hop IS the leak; the host now gets a decision point
b Parent always resumes after failed handover (old-code copy can ladder-drift into a second zombie) Parent exits when another LIVE bili instance owns P ("upgrade landed elsewhere"); resumes only for foreign non-bili squatters Resuming an old-code copy reproduces the same zombie class one level up
c child.kill() fire-and-forget Kill verified (2 s budget); survivor escalated with manual-cleanup hint Unverified kills are how orphans survived their parent's timeout
d Readiness = raw TCP answer on P Readiness = the replacement's own registry record (pid + actual port); TCP demoted to fallback for pre-registry children TCP cannot attribute the answer to our child
e Unmarked lane'd spawn racing a live same-build twin hops past it (P+1 idles, the serving twin becomes the orphan — the 2026-10-13 recurrence) Exits 1 loudly; the serving instance wins; the next launch attaches to it Same leak shape, second trigger path; deterministic older-wins rule without cross-process kill authority
f Marked replacement could move +1 off its just-bound port in the post-bind claim check (sockets tolerating concurrent bind) Post-bind ladder denied to marked replacements Second path where a replacement could hop

config surface

One new internal env var, resolved through the existing env-var convention (spawn-channel category, like BILI_LAUNCH_TOKEN / BILI_STRICT_PORT):

  • BILI_UPGRADE_REPLACEMENT=1 — set automatically by performSelfRestart on the replacement's spawn env only; never meant to be set by users or hosts. Documented in CONFIGURATION.md en+zh via the generator seed (website/config-reference/env-only.yaml; blocks regenerated with tools/gen-config-docs.mjs generate). No config-file field, no CLI flag, no default change anywhere else.

Tests

  • tests/listen-policy.test.ts (new) — full policy matrix: twin collision outranks replacement/strict/ladder (and applies token-independent); replacement never ladders (incl. exhausted wait budget); legacy launcher-child, strict-port, and manual-launch behaviors unchanged when no twin holds.
  • tests/auto-restart.test.ts — marker env carried without mutating the parent env; misplaced replacement terminated + service resumed + child verified dead; supersede-exit path (no resume, clean finish); unkillable replacement escalated loudly.
  • tests/port-zone-live.test.ts — live end-to-end (CI lane ACP_TEST_CLAUDE_NATIVE=1): (1) marked replacement against a squatted zone base exits 1 and nothing lands on base+1; (2) unmarked lane'd spawn against a LIVE same-build twin exits 1, nothing lands on base+1, and the twin keeps serving — a direct replay of the 2026-10-13 20:12 event.
  • Local targeted runs green on the rebased head: listen-policy + auto-restart 21/21; adjacent port-zone + instance-governance + wire-contract 51/51; typecheck clean. Full suite left to CI per local-test discipline.
  • Branch rebased onto current master (absorbs the fix: unblock master lanes — restore personaNamespace import + storage-render goldens after #3061 #3067 merge); conflicts resolved as union of intent (this PR's twin rule + the branch's post-bind/readiness follow-ups).

Out of scope (owner-decision follow-up)

@github-actions

github-actions Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-13_fix-3044-replacement-port-hop (b37e99e)

Option A — Install this exact build (recommended)

npm install -g billion-context@0.0.3644-pr.3055.61026d5

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr3055.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷
(ai: vllm-qwen/qwen3.8-27b)

冲突已解决并重新推送,PR 现在 mergeable: true(head 2a86aca08,基于 rebase 时的 master 623d3078c)。rebase 过程中按规范做了独立审核,发现并直接修掉了 2 个跨分支问题(各一个标注 commit,见下)。本地定向验证全绿;远程 CI 正在新 head 上跑,红了我会跟进。

Rebase 与冲突:master 在 PR 切出后前进了很多(含同域新机制 #2947 port-owner-guard、#2945 restart 隧道强关)。3 处文本冲突全部是「双方各自新增」的意图合并:CHANGELOG 条目位置(#3044 条目置顶,保留 master 全部条目)、src/restart.ts 的 SelfRestartDeps 类型(保留 #2895 的 destroyLingeringSockets + #3044 的 killVerifyMs)、测试文件 import 区(两组 import 都留)。src/server.ts 等自动合并后逐段核过语义。

审核中发现并已修的 2 个问题(不修则本 PR 的保证不完整):

  1. 第二条跳端口路径没堵上(commit 61cbc4a1c):fix: verify sole port ownership after bind; guard proxy-origin against clobbering a live holder (#2941) #2947 的 bind 后所有权检查(resolvePortClaim)会给任何带 launch token 的子进程「占位冲突时 +1」的权利,而自重启替换进程继承父进程的 BILI_LAUNCH_TOKEN——在容忍并发 bind 的 socket 语义下,它 bind「成功」后仍可能跳到 P+1,autoRestartOnUpdate + occupied target port -> orphan process leak: new instance silently hops to the next port, old instance never exits #3044 的孤儿形态经第二条路径复活。已加 postBindCanLadder()(同一纯策略模块,矩阵测试钉住)。
  2. 就绪检查把父进程自己的注册记录误读为「接管者」(commit 2a86aca08):新交接验证的外来 peer 搜索只排除了子进程 pid,但垂死的父进程自己在本端口有一条「仍存活」的注册记录——健康但尚未 announce 的替换进程会被判成「被我们取代」而遭 kill、旧服务恢复。master 的 [Bug] autoRestartOnUpdate 在 HTTPS_PROXY/MITM 接入下几乎必然失败:closeAllConnections() 关不掉 CONNECT 隧道 → "connections stuck after forced close" #2895 隧道交接测试(真 bili 服务器)当场抓到:superseded-by-peer 而非干净交接。已加 e.pid !== process.pid 排除(终局判定本来就有,就绪环漏了)。

结构化八问

  1. 是什么问题:自动升级时,替换进程拿不到目标端口就静默跳到 P+1/P+2…,变成没人路由到的闲逛孤儿(原报告:9 小时 14 个孤儿、~8GB 私有内存,stale 永不收敛)。本次任务:解该 PR 的合并冲突 + 审核。
  2. 什么原因导致的:三层叠加——① 没人标记「这是必须占住此端口的替换进程」,它继承了启动器的 EADDRINUSE 爬梯权;② 父进程用裸 TCP 探测判断就绪,无法区分应答来自自家孩子还是抢跑的对端;③ kill 是 fire-and-forget、失败后恢复服务又走同一条爬梯,旧代码副本也能漂移成僵尸。
  3. 影响范围与严重程度:所有开 autoRestartOnUpdate 且升级窗口内端口有争用的车道(旧实例未放完、宿主竞态重启)。旧行为下每个升级周期漏一个空闲孤儿且升级永不落地;老实例仍在服务,所以是资源泄漏 + 升级不收敛,不是不可用。
  4. 怎么修复的:spawn 时用内部 env 标记替换进程,它必须独占被释放的精确端口(保留 port-zone 重启竞态:sticky 端口单调漂移,升级依附与无界自增互相矛盾(#1660 后续) #1723 对慢释放前代的有界等待),否则 exit 1 响亮失败;父进程改按孩子自己的注册记录(pid + 实际绑定端口)验证交接——绑错端口→kill 并验尸→恢复服务;另一个活着的 bili 实例已占该端口→视升级已在别处落地、干净退出(不再恢复会漂移的旧代码副本);SIGTERM 杀不死→响报错提示手动清理。外加上面两个审核修复堵住第二跳路径与自我误读。
  5. 是否回归问题:逐项过过既有行为——非替换进程的 EADDRINUSE 全部路径(wait-predecessor / +1 爬梯 / 末次 ephemeral / strict-port / 手动启动 fail-fast)与旧内联处理逐分支等价(policy 矩阵测试钉死);外来非 bili 占用者保持原有 resume 行为;[Bug] autoRestartOnUpdate 在 HTTPS_PROXY/MITM 接入下几乎必然失败:closeAllConnections() 关不掉 CONNECT 隧道 → "connections stuck after forced close" #2895 隧道钩子、多实例共享 sessions 目录的重启竞态:丢失更新、前缀缓存击穿与会话跨实例暴露(#394 扩展) #1724 pre-spawn flush、Windows auto-update self-restart reports success but the replacement dies with the parent (spawn uses stdio:inherit without detached; bili stays offline after upgrade) #2095 平台选项矩阵 rebase 后全过。有意行为变更披露:替换进程 EADDRINUSE 由「爬梯到 ephemeral」改为「fail-fast exit 1」;另一 bili 实例占着端口时父进程由「总是恢复服务」改为「干净退出」(非 bili 占用者仍恢复)。
  6. 是否破坏缓存:不涉及(无缓存/重试/增量游标/去重键触碰)。
  7. 是否改了设计方向:不涉及——无架构转向;仅把决策表抽成纯函数(src/listen-policy.ts)以便钉测,与本 PR 自身及仓库既有模式一致。
  8. 是否引入新配置:是,1 个内部 env:BILI_UPGRADE_REPLACEMENT(boolean-ish,默认 unset,仅 performSelfRestart spawn 时设置,用户无需手设),CONFIGURATION.md 中英 + env-only.yaml seed 均已登记。

本地证据:tsc --noEmit 干净;定向套件 listen-policy / auto-restart / port-owner / port-zone-live 共 53 项 45 过 8 跳过(live 门控项走 CI 车道),0 失败。全量回归以远程 CI 在新 head 上的结果为准。master 随后合入的 #2731(google lane persona,只动 src/server/handle.ts + 2 个测试文件)与本 PR 零文件重叠,无需再 rebase。

收工判定:冲突已解、可合并状态、定向验证绿;等 CI 全绿后请人工合并(我不代合并):#3055

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

CI triage 完成:之前的 3 类红全部归因——一类是我们自己的(已修),另两类是 master 今日合入的 #2731 自己坏了,已单独立 issue。

① 我们的:unused-exports 门禁(已修)
新文件 src/listen-policy.ts 导出的 EaddrinuseAction 类型没有任何外部引用者,被 unused-exports 门禁判为 INTERNAL_ONLY。已改为模块内类型(commit 5d635100f,行为零变化)。本地门禁+测试+typecheck 全绿,CI 上该门禁已转绿确认。

② 不是我们的:master 自 #2731 起编译就是红的(已立 #3064)
今日合入的 #2731(fa7dc4e1b)在 src/server/handle.ts:811 调用了 personaNamespace(...) 但忘了 import(函数本身在 src/persona-anchor.ts:154 正常导出)。我在 master 的独立 worktree 里实测复现:npm run typecheck → error TS2304: Cannot find name 'personaNamespace'。tsup/esbuild 打包不做类型检查照常出包,产物里留下一个自由变量引用——google+dshPersona 代码路径一旦执行即运行时 ReferenceError(按 #2731 说明 dsh 今日不路由 google,暂时休眠,但接线已落地)。

为什么 master 自己的 CI 没抓到:push 事件的工作流只跑静态守卫,test/static-gates/typecheck/e2e 车道全部 skipped;而 PR 车道 checkout 的是「head 并入当前 master」的合并树(作业日志可见 Merge 2a86aca… into fa7dc4e1b…)——所以第一个撞上的不是 master 自己,而是本 PR 这类在途 PR。具体表现:ci-image 的 Typecheck 步红(TS2304)、dsh plugin lane 的 google-lane e2e 502 !== 200(正是 #2731 新增的那条接线路径)、static-gates 四格全红(同因)。

处置:已立 #3064(带来源标记),给出两个修复方向请 owner 拍板(补一行 import,或让 google lane 像其余三条 wire 一样走 session-resolve 的 wiring adapter),外加一条改进项(push 事件至少跑一次 typecheck,否则同类漏 import 会继续静默进 master)。handle.ts 的修法没有混进本 PR(保持 diff 范围干净);这几条车道要等 master 经 #3064 修好后自然转绿,本分支无需重推。

当前 head 5d635100f CI 快照:35 个 check-run,25 绿;剩余 6 红全部是上述 master 因(ci-image、dsh plugin lane、static-gates×4);4 个 test 全量车道仍在跑,预期绿(其文件清单不含受影响的 e2e 文件,且 tsx 不做类型检查)。

结论不变:本 PR 代码侧已完成(rebase 到当前 master + 两处评审修复 + 门禁修复,共 4 个干净 commit),等 #3064 在 master 修好、CI 全绿后人工合并。

ework-agent added 4 commits October 14, 2026 00:20
… of hopping (#3044)

With autoRestartOnUpdate, a replacement spawned while the old instance still
held its port inherited the launcher's EADDRINUSE ladder and silently drifted
to P+1, P+2, ...: idle orphans serving no client while the prior instance
never exited (#3044 reported 14 orphans / ~8 GB private memory over 9 h).

- New src/listen-policy.ts: pure EADDRINUSE decision table. A spawn marked
  BILI_UPGRADE_REPLACEMENT=1 (set by performSelfRestart at spawn time) keeps
  the bounded same-lane predecessor wait (#1723) and then fails fast instead
  of laddering. All non-replacement paths are byte-identical to the old
  inline handler.
- performSelfRestart verifies the handover against the replacement's own
  registry record (pid + port it actually bound) instead of a raw TCP answer:
  wrong port -> terminate + resume; another live bili instance owns the port
  -> treat the upgrade as landed there and exit cleanly instead of resuming
  an old-code copy that would ladder-drift into its own zombie; foreign
  non-bili squatters keep today's resume behavior. Kill results are verified
  and a surviving child escalates loudly with a manual-cleanup hint.

Tests: tests/listen-policy.test.ts (policy matrix), auto-restart.test.ts
(marker env, misplaced termination, supersede-exit, unkillable escalation),
port-zone-live.test.ts (live fail-fast end-to-end, CI lane). Internal env
BILI_UPGRADE_REPLACEMENT documented in CONFIGURATION.md en+zh via the
generator seed.
… x #2947)

Cross-branch gap found while rebasing #3044 onto master: #2947's post-bind
port-ownership check (src/port-owner.ts resolvePortClaim) gives any
launch-token child the right to move +1 off a contested just-bound port. A
self-restart replacement inherits its parent's BILI_LAUNCH_TOKEN, so on
socket semantics that tolerate a concurrent bind onto an actively-listening
port it could still hop P+1 after a 'successful' bind — the exact orphan
shape #3044 kills, via the second hop path. The EADDRINUSE handler was
already closed by this branch; this closes the claim path too:
postBindCanLadder() in the same pure policy module, pinned in
tests/listen-policy.test.ts.
…#3044 x #2895)

The new handover verification's foreign-peer search excluded only the
child's pid. The dying parent bili process has its OWN registry record for
this very port (written at its announceListening), still 'live' while we
drain — so a healthy replacement that bound the port but had not yet
announced read as superseded by us and got killed, resuming old service.
Master's #2895 tunnel-handover test (real in-process bili server) pins it:
superseded-by-peer instead of a clean handover. Exclude process.pid, the
same way the terminal decision already does.
EaddrinuseAction is only referenced inside src/listen-policy.ts (decideEaddrinuse's
return type); nothing imports it, so the unused-exports gate flagged it
INTERNAL_ONLY. Make it module-local. Behavior unchanged.
@ranxianglei
ranxianglei force-pushed the 2026-10-13_fix-3044-replacement-port-hop branch from 5d63510 to b33ca09 Compare October 13, 2026 16:24
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

上游冲突已处理:rebase 到最新 master(c800ed1bb),强推完成,新 head b33ca0980。

这次发生了什么:上一轮 rebase 之后 master 又合入了 4 个 PR(#2731 后续、#3058 kernel preserveRecentMessages 5→3、#3056 codex compact、#3052 dsh exports、以及关键的 #3067),PR 再次与基线冲突。

重要发现——上一轮的满屏红其实都是 master 自己的伤,且已全部修好:#3067(「unblock master lanes」)一次性修了两件事——a9378038a 补回 #2731 丢掉的 personaNamespace import(#3060,即 static-gates/ci-image Typecheck/dsh-lane 的 TS2304 与 502 根因)+ 8f93c33e3 重生成被 #3061(preserveRecentMessages 5→3)打坏的 storage-render goldens(test×4 车道红的根因)。这两个修复现在都在本分支树里,rebase 后本地 typecheck 由红转绿即为直接证据。我此前立的 #3064 与 #3060 判定一模一样(同症状同机制,#3060 证据更完整),已按查重规则关闭并打 duplicate 标签,评论里留了说明。

结构化八问:

  1. 是什么问题:上游冲突——master 在我们 rebase 后又前进了 4 个合并,PR 与基线再次冲突。
  2. 什么原因导致的:并行开发的正常漂移;本次三个重叠文件(src/server.ts、CONFIGURATION.md/zh)均自动合并成功,无文本冲突。
  3. 影响范围与严重程度:无——纯 rebase,diff 内容零变化,只有基线前进。
  4. 怎么修复的:rebase 到 c800ed1bb 后对三处自动合并做了语义核对(union of intent):我们的 BILI_UPGRADE_REPLACEMENT 文档行、postBindCanLadder 调用点、CHANGELOG 条目全部保留,master 的 preserveRecentMessages=3 新行也在;本地 typecheck 干净 + 定向测试 45 过 / 0 挂(8 个 live-gated 跳过,CI 跑)。
  5. 是否有回归问题:本次未引入任何行为变化;顺带确认了上一轮 8 条红车道(static-gates×4、test×4、ci-image、dsh-lane)全部归因于 master 侧 master typecheck broken: handle.ts:811 calls unimported personaNamespace (google dsh persona path) #3060/compress.preserveRecentMessages default 5 → 3 (#3058) #3061 两处破坏,均已随 fix: unblock master lanes — restore personaNamespace import + storage-render goldens after #3061 #3067 修复进入我们的树——预期本轮 CI 全绿。
  6. 是否破坏缓存:不涉及。
  7. 是否改了设计方向:不涉及。
  8. 是否引入新配置:不涉及(BILI_UPGRADE_REPLACEMENT 为原 PR 的内部 spawn 通道,已按 en+zh 文档化,未变)。

判定:代码侧就绪,等本轮 CI 全绿后人工合并。

awork and others added 2 commits October 14, 2026 20:33
… of hopping (#3044)

Second trigger path from the 2026-10-13 recurrence on 0.1.197: a host
respawn (dsh plugin channel) races an attach scan and spawns while an
identical instance already serves the lane's port; that child laddered to
P+1 and idled as an orphan while traffic re-pointed to it.

findSameLaneTwin recognizes a live same-lane SAME-build holder (the mirror
of findSameLanePredecessor, which only matches different builds). Any
non-strict lane'd spawn hitting it now fails fast with exit code 1 and a
loud log line instead of laddering: the duplicate loses, the serving
instance keeps serving, and the next launch attaches to it. The rule is
deliberate and deterministic — tearing down a live-serving instance for an
identical fresh one buys nothing, and no admin endpoint or cross-process
kill authority is needed. Strict-port and manual launches are unaffected
(they already fail fast).
@ranxianglei
ranxianglei merged commit bd2aa6e into master Oct 14, 2026
27 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

📦 Merged Build Published

This merge to master was published as 0.0.419-master.bd2aa6e (rolling master tag).

npm install -g billion-context@0.0.419-master.bd2aa6e  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

ranxianglei pushed a commit to El1mir/billion-context that referenced this pull request Oct 14, 2026
Follow-up to the conflict takeover: master advanced (ranxianglei#3055 port-hop
fixes, ranxianglei#3165 per-lane attach policy, ranxianglei#3072 codex WS egress fidelity)
after the first merge. Conflicts confined to files the branch never
semantically touched; resolved verbatim to origin/master.
@ranxianglei
ranxianglei deleted the 2026-10-13_fix-3044-replacement-port-hop branch October 15, 2026 05:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

autoRestartOnUpdate + occupied target port -> orphan process leak: new instance silently hops to the next port, old instance never exits

1 participant