Skip to content

[PR] fix: verify responder identity before claiming a settled proxy origin (#4125) - #4622

Open
ranxianglei wants to merge 12 commits into
masterfrom
2026-10-25_4125-settle-identity
Open

ranxianglei wants to merge 12 commits into
masterfrom
2026-10-25_4125-settle-identity

Conversation

@ranxianglei

@ranxianglei ranxianglei commented Oct 24, 2026 •

Copy link
Copy Markdown
Owner

Related to #4125 (partial slice — the root cause of the multi-instance episode still awaits the reporter's log evidence requested in-thread). This PR now holds three facets of the same bootstrap-failure mechanism family; each is its own labeled commit.

Facet 1 — verify responder identity before claiming a settled origin (commit 3ededa8)

ensureProxyRunning's settle loop has three branches that can declare "our child is up":

  1. launchToken match — instance file carries our spawn token and its origin answers health;
  2. registry-by-pid — registry entry with our child's pid, origin answers health;
  3. stale-record fallback — already verified the responder pid equals our child's pid since launcher: spawn 等待的 pre-handshake fallback 把首选端口上的健康外来实例当成自己 spawn 的 child,客户端被 export 到错误 origin(绕过 #1225 指纹门) #1753.

Branches 1 and 2 trusted any healthy answer at the claimed origin. If a foreign listener positively answers /__bili/health there with a different live pid, the old code handed that origin out as ours (single-writer violation, #991/#1225) or burned the full 60 s spawn budget probing a contested port and then killed its own child — one concrete way bootstrap fails for hours while another instance keeps running, which is the shape #4125 reports.

Both primary branches now use fetchHealthInfo and accept only when the reported pid is absent (legacy fail-open preserved for older children) or equals the spawned child's pid. A positive foreign-pid answer is rejected with a once-only diagnostic naming both pids. Terminal paths (kill + diagnosePort) are unchanged — they already name this situation at decision time.

Also corrects a stale citation in src/port-owner.ts: it pointed at libuv issue #1360 (an unrelated uv_get_process_title segfault fix) for the Windows bind-exclusivity claim; the actual basis is in libuv source (src/win/tcp.c, uv__tcp_try_bind): libuv deliberately omits SO_REUSEADDR on Windows because setting it would allow stealing a port held by another application.

Facet 2 — fresh owner heartbeat vetoes transient kill-0 failures in the parent-gone watchdog (commit aebbafb)

Field evidence from #4125 (DSH web host, 2026-10-25): parent-gone (pid 14852) received fired against a LIVE host and forced a healthy, mid-compression proxy to exit ~10 s later. The quoted shutdown reason carried no [stale-heartbeat pids: …] suffix — which the code appends ONLY for heartbeat-staleness removals — so the false death ran the kill-0 confirmation path: three consecutive transient OpenProcess failures against a live pid burned the whole #3890 confirm window (~4–6 s) plus the idle grace.

The watchdog now has two liveness signals: the identity-blind kill-0 probe and the identity-anchored owner heartbeat (only the true owner posts its own pid over loopback). A failed probe on a heartbeat-opted watcher whose last beat is within the heartbeat grace window (15 s) no longer counts towards the death streak; the disagreement is logged once per episode with the probe errno. Stale-heartbeat removal (#3544) and the legacy kill-0-only semantics for non-heartbeat pids are unchanged (both pinned).

Behavior change: a truly dead parent whose last heartbeat landed moments before death now keeps the proxy alive up to heartbeat grace (15 s) + idle grace (5 s) instead of ~11 s — a bounded orphan-lifetime extension in exchange for not killing live sessions on transient probe flaps.

Facet 3 — scope spawn-log evidence to this spawn + record the at-spawn port state (commit 4537b63)

The port log (%TEMP%/bili-proxy-<port>.log) is shared across spawns (append fd, size-based rotation only). A zero-output failed child left the PREVIOUS spawn's content behind, and every quoted sample misattributed it — field case: a week-old EADDRINUSE line quoted as the current failure's evidence. And nothing recorded what the port was doing at SPAWN time, while the failure-time diagnosis only sees the END state — "predecessor still draining" and "child wedged elsewhere" read identically at the end.

Each spawn now appends a === bili spawn <ts> launcherPid=<pid> childPid=<pid> port=<p> === header; size reports and head samples are scoped to this spawn's segment ("N bytes since this spawn"); every terminal failure message (slow-start notice, early exit, timeout throw + console.error) carries (at spawn: <port state>) — healthy bili instance (pid N) / foreign HTTP answer / nothing listening / HTTP nnn / busy (TCP up, no health answer). Rotation policy and on-disk content are otherwise unchanged.

Also fixes a test-fake gap this PR exposed: the #3544 bring-up harness modeled the spawned child through the bare-probe seam but not the health-info seam, so facet 1 starved its settle loop; the fake now models a real child whose /__bili/health names its own pid.

Branch note

Head builds on ff3564d4b (sentinel help-merge bringing ci-fixes into this branch per the #4599 inherited-red policy); facets 1–3 were rebased onto it cleanly.

Behavior change disclosure (all facets)

  • Facet 1 — old: settle loop accepted any healthy /__bili/health response at the child's claimed origin (branches 1–2). New: a response that positively reports a different pid is rejected (loop continues, once-diagnostic logged); absent-pid responses behave exactly as before.
  • Facet 2 — old: every failed kill-0 probe counted toward the death streak regardless of heartbeat freshness. New: vetoed while a fresh heartbeat exists (bounded orphan-lifetime extension as above).
  • Facet 3 — additive fields in failure/diagnostic messages ((at spawn: …)); empty-log wording changed from 0 bytes — EMPTY, the child produced no output of its own to 0 bytes since this spawn — EMPTY …; per-spawn header lines appended to the shared port log.
  • No wire/config/persistence/env surface touched in any facet.

Tests

…#4125)

The ensureProxyRunning settle loop trusted any healthy /__bili/health answer at the child's claimed origin in its two primary branches (launchToken match, registry-by-pid), while only the stale-record fallback verified the responder pid (#1753). A foreign listener positively answering with a different pid therefore counted as success: the launcher would hand that origin out as our own (single-writer violation) or burn the full spawn budget probing a contested port.

Both primary branches now use fetchHealthInfo and accept only when the reported pid is absent (legacy fail-open preserved) or equals the spawned child's pid; a positive foreign-pid answer is rejected with a once-diagnostic naming both pids. Terminal paths unchanged. Also corrects the stale libuv-issue citation in port-owner.ts (libuv omits SO_REUSEADDR on Windows precisely because setting it allows stealing another application's port).

Tests: tests/issue4125-settle-identity.test.ts pins accept/reject/fail-open per branch (both gate directions, #3617 discipline); launcher suite green.
@ranxianglei ranxianglei changed the title fix: verify responder identity before claiming a settled proxy origin (#4125) [PR] fix: verify responder identity before claiming a settled proxy origin (#4125) Oct 24, 2026
@github-actions

github-actions Bot commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-25_4125-settle-identity (ebb638d)

Latest build — install this exact version

npm install -g billion-context@0.0.7498-pr.4622.398f227

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Recent builds (commit → version)

commit version build
ebb638d billion-context@0.0.7498-pr.4622.398f227 build
6d167d1 billion-context@0.0.6168-pr.4622.6a56edf build
4b7d570 billion-context@0.0.6081-pr.4622.7415e6e build
1f62504 billion-context@0.0.5984-pr.4622.ed9a36b build
ff071b3 billion-context@0.0.5876-pr.4622.932b0a5 build
34581e3 billion-context@0.0.5809-pr.4622.5a6df96 build
9715110 billion-context@0.0.5701-pr.4622.48e0cbe build
4537b63 billion-context@0.0.5700-pr.4622.3620f04 build
ff3564d billion-context@0.0.5649-pr.4622.6b2836e build
4939841 billion-context@0.0.5600-pr.4622.806b9f1 build

Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr4622.tgz
npm install -g package

Install instructions are refreshed on each push.

…ed-exports

#4545's dsh-lane-only trim removed every external use of the type; it is now
referenced only inside client-build-note.ts, which trips the INTERNAL_ONLY
gate on clean master (reproduced locally). Drop the leading export per the
gate's prescribed fix — a future lane re-adds it when it actually opts in.
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重:不重复——本次红线的根因是 master 上遗留的僵尸导出,master 侧已有专门修复在途(#4606,sentinel 在 #4599 接管);按「几行级小修」政策在本 PR 分支直接顺手修(独立标注 commit),跨 PR 重复修复明确允许。

结构化八问:

  1. 是什么问题:CI 的 unused-exports 检查红——src/agent/client-build-note.ts 里的 ClientBuildCheck 类型声明了 export,但全仓库再没有任何文件在文件外引用它。
  2. 什么原因导致的:不是本 PR 引入的。[PR] dsh lane: note in-session when running an unverified @deepseek-ai/dsh client build (soft, exact-match) #4545 把「客户端新版本提示」功能收窄到只剩 dsh 车道时,删掉了所有外部对该类型的引用,在 master 上留下这个僵尸导出——master 自身从那时起就红着(clean master 本地跑同一检查脚本复现出一模一样的报错),本 PR 只是继承了这个红线。
  3. 影响范围与严重程度:只挡 CI 门禁(任何 PR 合并都会被它挡住,包括本 PR);运行时行为零变化、用户零感知。严重度 = 构建阻断级,无外部影响。
  4. 怎么修复的:去掉该类型声明的前导 export(一个词,正是检查脚本报错信息自己给出的修法)。类型在文件内部照常使用;将来第二个车道真要接入时再把 export 加回来即可。没有走 WAIVERS 豁免名单——「以后可能用」不满足豁免标准。
  5. 是否有回归问题:无。逐项清点改动触碰路径的既有行为:该文件导出的五个成员(maxTestedVersion / walkUpNodeModuleVersion / DSH_BUILD_CHECK / newerClientBuildMessage / emitNewerClientBuildNote)全部保持导出、签名不变;全仓检索确认没有任何文件从外部 import ClientBuildCheck。无阈值/时序/输出格式漂移,fail-fast 语义未动。
  6. 是否破坏缓存:不涉及——纯编译期类型可见性变更,不碰任何运行时、缓存或去重键。
  7. 是否改了设计方向:不涉及。
  8. 是否引入了新的配置:不涉及。

修复已推送:commit 493984166(一行 diff),本地检查脚本通过(✓ unused-export gate: clean)。备注:master 上同一僵尸导出仍在,master 侧根治在 #4606,其合入后本 PR rebase 时这一行 diff 会自然收敛。等 CI unused-exports 道重跑结果。

@ranxianglei ranxianglei added bug Something isn't working dsh p1 scope/wiring labels Oct 24, 2026
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重:不重复(同机制家族、不同位置)。 本 PR 把 #1753/#1758 早已确立的「认领某个 origin 前先核对应答方 pid」纪律,从 spawn-wait 的 preferred-port 兜底分支,补齐到 settle 循环的两个主分支(launchToken / registry-by-pid)。相关既有工作:#1758(spawn-wait 兜底)、#2947(bind 后端口所有权)、#1756(launcher 生命周期两侧)——都属同一 responder-identity 机制家族,各修各的位置。本 PR 补的是「主分支一直没对齐到 #1753 标准」这个真实缺口,不是重复劳动。

标签: bug + dsh + scope/wiring + p1 —— 修的是单写者/可用性误路由(p1 级机制缺陷的部分切片),宿主来自 #4125 的 dsh 桌面车道。


1. 是什么问题:settle 循环的 launchToken 与 registry-by-pid 两个主分支,原来只要「child 声称的那个 origin 上有人健康应答 /__bili/health」就当成自己的子进程导给客户端,完全不看应答方的 pid。若那个地址被另一个活着的 bili(旧版本残留 / 另一车道)占着并健康应答,旧代码会把别人的实例当自己的交出去(违反单写者 #991/#1225),或在整个 60s spawn 预算里反复探一个被占的端口、最后杀掉自己的孩子。这正是 #4125「bootstrap 起新实例、探活失败、goes direct」的一个具体成因。

2. 什么原因导致的:根因是把「健康应答」当成了「身份证明」。preferred-port 兜底分支在 #1753 就吃过亏、改成必核 pid,但两个主分支一直沿用旧的宽松判定(只看布尔 probeHealth)。身份证据其实早就在手(launchToken 每次 spawn 全新随机、registry 记录里带着 child 的确切 pid),缺的只是拿它去对一下应答方报的 pid。

3. 影响范围与严重程度:触发条件 = 同一地址上存在第二个活着、能健康应答且带不同 pid 的实例(多实例 / 旧版本残留 / 跨车道混淆)。严重度 = p1 级机制缺陷:命中时要么静默误路由(用户侧完全看不到,正是 #4125 排障成本高的根源),要么烧满 60s 预算后杀自己孩子、bootstrap 长时间失败。本 PR 只覆盖 settle 循环这一处切片,#4125 的多实例根因仍待 reporter 的日志证据(原线程已索取)——PR 正文也如实标注 partial slice。

4. 怎么修复的:两个主分支改用 fetchHealthInfo 拿到带 pid 的健康信息,只在「应答方没报 pid(老构建,fail-open 保持旧语义)」或「pid == 自己 child 的 pid」时才认领;若应答方明确报了别的 pid,拒绝认领并打一条一次性诊断(同时点名两个 pid)。终态路径(kill + diagnosePort)不动,本来就在决策点说清这种情况。关键设计点:主分支握有强身份凭据(token / pid 记录),所以保留 absent-pid 的 fail-open;而 #1753 的 preferred-port 兜底没有任何凭据可托,才要求严格 pid 相等——这个不对称是刻意的、自洽的,不是疏漏。

5. 是否有回归问题:审过的既有行为逐项——

  • own-child 永远报自己的 process.pid,即等于 child.pid,不会被误拒(这条安全性质成立,是整个修法的地基);
  • absent-pid 应答的行为与旧代码一致(fail-open 保留);
  • launchToken 每次 spawn 都是全新随机值,跨 spawn 不复用,不存在陈旧 token 误判;
  • registry 分支本就包在 child.pid !== undefined 内,新增判定不破坏该前提;
  • 无 wire / config / persistence / env / kernel 面改动。
    唯一微小代价:失败路径每轮 poll 会先 fetchHealthInfo 再 fallback probeHealth(两次探测)。这不是纯冗余——2xx 但 JSON 解析失败的退化应答会走 fallback 保住旧语义;代价仅是立即拒绝/快速响应的连接,60s 内约百次量级,可忽略。高危模式逐项过:无语义漂移、无吞错(诊断反而更响亮)、无阈值/时序/输出格式变化。结论:无回归。

6. 是否破坏缓存:不涉及(纯 launcher/bootstrap 路径,不碰前缀缓存 / 压缩 / 存储游标)。

7. 是否改了设计方向:不涉及架构转向,是把既有 #1753 身份纪律在同类位置补齐,方向一致。附带把 src/port-owner.ts 一处过期引用改对:原指向 libuv issue #1360(那是无关的 uv_get_process_title segfault 修复),Windows bind 独占性的真正依据是 libuv 源码 src/win/tcp.c 的 uv__tcp_try_bind 刻意不设 SO_REUSEADDR(设了就能抢走别人占着的端口)。

8. 是否引入了新的配置:不涉及(无新 env / config / 默认值)。


CI 状态:head 3ededa8c 上唯一红门 unused-exports 是继承自 master(base sha 146b34f2 同样红;#4599 总控已确认根因 = INTERNAL_ONLY src/agent/client-build-note.ts: ClientBuildCheck,修复在 #4603)。本 PR 只动 launcher.ts / port-owner.ts,零新导出,与该红无关;其余门禁全绿或按条件 skip。→ 等 #4603 落 master 后 rebase + retrigger 即转绿,不必为它排查。

测试有效性:tests/issue4125-settle-identity.test.ts 共 6 例,按 #3617 双向钉住每个分支——own-pid 认、foreign-pid 拒(诊断恰好一次、消息含两个 pid)、absent-pid 放行(fail-open)、health-info 无返回时回落布尔探针。断言具体(origin/port/reject 正则/诊断计数),非恒真。一个小缺口:branch 1 里 child.pid === undefined 的防御性 fail-open 未单独测(属刻意降级回旧行为的边缘,低风险,不强求)。

合并判定:可合。 建议 owner 在 #4603 清掉 master 红之后 rebase + retrigger 再合。提醒一句:这是 spawn-settle 身份逻辑,落在 load-bearing 的 bootstrap 路径上,按 §7「identity/session-binding logic」应留人工合入、不走 auto-merge。我作为 agent 不执行合入,以上为独立审核结论。

ranxianglei pushed a commit that referenced this pull request Oct 24, 2026
…ot this PR's

master 146b34f fails the unused-exports gate on this exact line
(INTERNAL_ONLY src/agent/client-build-note.ts: ClientBuildCheck): the
type-only export #4545 introduced was missed by the 4d34abf de-export
sweep. This PR touches none of it — every PR rebased past that master
inherits the red (#4599 storm log). Same one-word fix as #4606 commit 1
and the in-branch precedents (#4607/#4595/#4540/#4618/#4622): drop the
leading export on a symbol referenced only within its own file. Drop
this commit when rebasing after #4606 merges.
ci-check added 8 commits October 25, 2026 10:09
…rent-gone watchdog (#4125)

The watchdog had two disagreeing liveness signals: the identity-blind kill-0 existence probe (death streak of confirmTicks, #3890) and the identity-anchored owner heartbeat (#3544). The streak ran regardless of heartbeat freshness, so three consecutive transient OpenProcess failures against a LIVE host burned the whole confirm window and force-exited a healthy proxy mid-session. Field case 2026-10-25 (DSH web host, reported in #4125): parent-gone fired for a pid that was alive throughout; the shutdown reason carried no [stale-heartbeat] suffix, pinning it to the kill-0 path.

A FAILED probe on a heartbeat-opted owner no longer counts while its last beat is within the grace window: the beat is the stronger proof (only the true owner posts its own pid over loopback). The disagreement is logged once per episode with the probe errno. Stale-heartbeat removal (#3544) and legacy kill-0-only semantics for non-heartbeat pids are unchanged (pinned in tests).

Behavior change: a truly dead parent whose last heartbeat was delivered moments before death now keeps the proxy alive up to the heartbeat grace (15s) plus idle grace (5s) instead of ~11s — bounded orphan-lifetime extension in exchange for not killing live sessions on transient probe flaps.
…ort state (#4125)

The port log (%TEMP%/bili-proxy-<port>.log) is shared across spawns (append fd, size-based rotation only). A zero-output failed child left the PREVIOUS spawn's content behind and every quoted sample misattributed it — field case 2026-10-25: a week-old EADDRINUSE line quoted as the current failure's evidence. Nothing recorded what the port was doing at SPAWN time, while the failure-time diagnosis only sees the END state; 'predecessor still draining' and 'child wedged elsewhere' read identically at the end.

Each spawn now appends a '=== bili spawn <ts> launcherPid childPid port ===' header; size reports and head samples are scoped to this spawn's segment ('N bytes since this spawn'); every terminal failure message (slow-start notice, early exit, timeout) carries '(at spawn: <port state>)'. Rotation policy and on-disk content are unchanged — only the quoting and the new fields.

Also fixes a fake gap this PR exposed: the #3544 bring-up harness modeled the spawned child through the bare probe seam but not the health-info seam, so the settle-identity commit in this PR starved its settle loop. The fake now models a real child whose /__bili/health names its own pid post-spawn.

Behavior change: failure/diagnostic messages gain fields (additive); the empty-log wording changes from '0 bytes — EMPTY' to '0 bytes since this spawn — EMPTY'.
 heartbeat veto; publish spawn-log header pre-spawn

Merge resolution (origin/master into 2026-10-25_4125-settle-identity):
- tickWatchedPid: union of master's #4994 dual-axis signature (hbStreak + heartbeatConfirmTicks) and this branch's #4125 fresh-heartbeat veto; the veto resets hbStreak too, and the call site clears the kill-0 streak on a vetoed tick.
- spawn log: folded the branch's parallel spawnLogStartBytes into master's spawnLogOffset. The '=== bili spawn ===' header now publishes BEFORE spawnImpl (same discipline as #707's starting marker): a spawned child can write before a post-spawn append lands, which would push its earliest bytes (crash tails, listen lines) below the offset into 'previous spawn' territory. The header drops childPid (unknown pre-spawn); the child names itself in its pid-tagged listen line (#4994).
- terminal failure messages keep BOTH features: master's structured diagnosePort().text + failureHint() AND this branch's '(at spawn: <port state>)' annotation.
- tests adapted to the merged contracts: issue4125-watchdog-heartbeat-veto (6-arg signature + #4994 confirmed-tick pin replacing the immediacy pin), issue4125-spawn-log-segment (header asserts port= instead of childPid), issue4811 (zero-output wording pin).
Local: typecheck clean; all 12 affected suites green serially (82 tests).
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Merged current master into this branch and resolved all conflicts — the PR is mergeable again (head ebb638da6, state "behind" only because master kept moving; not a blocker). Resolution was union-of-intent throughout; nothing from either side was dropped. Three places where the two sides genuinely collided:

1. tickWatchedPid (src/server.ts) — master shipped #4994's dual-axis watchdog (kill-0 streak AND heartbeat-stale streak, tick-confirmed) while this branch added the #4125 fresh-heartbeat veto on top of the older single-axis signature. Merged contract: master's 6-arg signature (prevHbStreak + heartbeatConfirmTicks) with the veto inserted between the alive-branch and the kill-0 increment; a vetoed tick resets BOTH streaks (hbStreak: 0, call site deletes the dead-streak entry) so a healthy heartbeat burns neither confirm window. Pinned by the rewritten tests/issue4125-watchdog-heartbeat-veto.test.ts (veto tests kept verbatim in intent + a new #4994 confirmed-tick pin replacing the old immediacy pin).

2. Spawn-log segment (src/launcher.ts) — master independently built #4811/#4816 sampling on spawnLogOffset (captured at file-open) while this branch invented a parallel spawnLogStartBytes (captured after appending the === bili spawn === header). Keeping both variables meant two offsets describing one segment; folded everything onto master's spawnLogOffset. Merging exposed a real race in the post-spawn header design, reproduced locally: a spawned child can write its first bytes BEFORE the parent's post-spawn appendFileSync(header) lands, pushing exactly those bytes (crash tails, listen lines, first output) below the offset — they then get attributed to the "previous spawn" and the failure report says 0 bytes since this spawn — EMPTY even though the child talked. Under parallel test load this failed #5096's tail-sample tests, #4994's rung-2 listen-line adoption, and #4811's head-quote test. Fix: publish the header before spawnImpl (same discipline as #707's starting marker) so the boundary is physically guaranteed. Two disclosed behavior changes: (a) the header drops childPid= (unknown pre-spawn) — attribution is carried by launcherPid+port+timestamp plus the child's own pid-tagged listen line that #4994's adoption already parses; (b) the tail annotation now reports , N bytes since this spawn (this child's bytes only) instead of master's , N bytes (whole-file size) — with a shared append-only log, whole-file size is exactly the misattribution #4811/#4125 exist to stop.

3. Terminal failure messages — master restructured diagnosePort() to return {text, selfHealthy} and added failureHint(); this branch appended (at spawn: <port state>). Kept both: (…${(await diagnosePort(x)).text}; at spawn: ${initialPortNote})…${failureHint()}.

Tests adapted to the merged contracts: issue4125-watchdog-heartbeat-veto (signature + #4994 pin), issue4125-spawn-log-segment (header asserts port= instead of childPid=), issue4811-dsh-busy-drop-selfheal (zero-output wording pin). Local validation: typecheck clean, all 12 affected suites green serially (82 tests). Full-suite verdict is CI's on this head.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working dsh p1 scope/wiring

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant