Skip to content

[PR] fix: three-state health probe — a busy proxy is no longer misjudged as dead (#3654) - #3673

Merged
ranxianglei merged 3 commits into
masterfrom
2026-10-18_health-probe-three-state
Oct 19, 2026
Merged

ranxianglei merged 3 commits into
masterfrom
2026-10-18_health-probe-three-state

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

Closes #3654

What changes

The health probe now distinguishes three states instead of two:

verdict meaning action
alive /__bili/health answered {ok:true} within budget route through the origin
dead TCP connect refused (classified in ms), or answered but not healthy clear origin, respawn — the fast path is unchanged
busy TCP connect OK but no health answer within budget (event loop saturated, typically a large compression in flight) keep the origin, re-probe at ~1s jittered steps up to a 10s cap; only then let go and respawn

Key pieces:

  1. src/liveness.ts (new) — shared probeLiveness(origin, budgetMs) + tcpConnectable(). A raw TCP-connect pre-probe is what makes busy/dead discriminable in milliseconds: the kernel accept queue answers connects while the JS event loop is blocked (measured: 2ms connect during an 8s block), while a refused port yields ECONNREFUSED immediately.
  2. src/agent/native-intercept.ts — createLiveOriginResolver (both call sites: opencode native V2 route + opencode launcher route) consumes the verdicts: busy holds the origin without spawning (no cooldown bypass, no generation rebuild); the TTL cache stores alive/dead only, so busy is never cached as dead (per the [Bug][OpenCode] 同 origin 代理重启复用旧负健康缓存,恢复首请求绕过压缩 #1957/fix(#1957): invalidate stale health verdicts across the respawn boundary #1966 generation discipline); a replacement instance judged busy is routed to anyway (the request queues behind the fold); resolver failures record lastUnavailableReason for downstream wording.
  3. Spawn-side self-heal (src/launcher.ts, src/server.ts, src/listen-policy.ts, src/instance.ts) — a spawned child that loses the autoRestartOnUpdate + occupied target port -> orphan process leak: new instance silently hops to the next port, old instance never exits #3044 twin race now exits with code 3 (was 1) as an explicit adopt signal; the spawner verifies the serving instance (registry identity + live pid + TCP reachability) and attaches to it instead of failing the whole bootstrap. This removes the "discovery says dead / twin rule says alive" contradiction instead of de-frequencing it.
  4. Diagnostics — the per-port log file receives the listen-failure line (the $TMPDIR/bili-proxy-<port>.log that was exactly 0 bytes in [Bug?] Under high load, health check timeouts appear to be misinterpreted as "proxy unavailable": large session requests | [Bug?] 高负载下 health 探活超时疑似被当成「代理不可用」:大会话请求被本地拒发(0.1.199 / plugin native) #3654); the discovery drop and the port diagnosis now report "BUSY (event loop saturated)" when TCP connect succeeds; every classification lands in bili.log with attempt counts.
  5. Wording split — the saturated-proxy refusal no longer claims "guaranteed upstream 400": it states the proxy is alive but unresponsive and likely to serve once the fold finishes. The legacy wording stays byte-identical for a genuinely unavailable proxy.

Behavior changes (old → new)

# change old new why
① probe timeout against a live-but-saturated proxy instant death → drop origin → respawn → twin-collision → possible local refusal busy → hold origin, jittered re-probes (~1s, 0.5×–1.5×) up to a 10s cap → then let go + respawn + adopt the existing instance the misjudgment was the bug: with p99 compression at 25s no constant budget chases the tail, but "slow" ≠ "dead"
② detection latency for timeout-style death 1.5s ≤10s + spawn the cost of discriminating busy; the ECONNREFUSED fast path is unchanged (milliseconds)
③ "alive" verdict payload res.ok only {ok:true} JSON validation a foreign listener returning 200 on that path is not our proxy (parity with the launcher-side #1753 discipline)
④ twin-collision child exit code 1 (indistinguishable from a crash) 3 (adopt signal) lets the spawner tell "lost the duplicate race" apart from "crashed before becoming healthy"
⑤ busy replacement instance probe failed → undefined → direct/refuse routed to anyway refusing here would defeat adoption: the instance is verified alive, the request just waits for the fold
⑥ refusal wording for a saturated proxy "proxy unavailable … guaranteed upstream 400" "proxy alive but unresponsive … likely to serve once the fold finishes" a saturated proxy is not "unavailable"; the old claim was false in this case

No new config/env surface (option C was declined in the #3654 thread). Kernel untouched.

Deliberately unchanged (parity notes)

  • The WS lane (native-ws-intercept.ts) has no probe to misjudge — readyOrigin only; it benefits indirectly through the bootstrap adoption.
  • Boot-time attach verification (shared.ts: 5s status timeout / 15s deadline) is unchanged — a boot misjudgment self-heals through spawn→adopt, which is now bounded by the same adoption path.
  • Reactive fetch-lane recovery (recover()) keeps its shape (clear + respawn on an actual send failure); it benefits through bootstrap adoption.
  • probeLiveInstances still drops candidates that fail the full attach gate (claude-native hooks must not attach to a lifecycle-less proxy — attach gate + self-managed fallback #1335) — now with a BUSY diagnostic attached.

Tests

New tests/issue3654-health-probe-three-state.test.ts (23 tests, hermetic, ephemeral ports via listen(0)):

  • probeLiveness against real sockets: alive / dead-fast (refusal classified <300ms) / busy (accepts, never answers) / foreign HTML 200 → dead / {ok:false} → dead
  • resolver contract: busy→alive inside the window (zero respawns, origin kept); busy→exhaust→respawn→routed; busy not cached across calls; dead cached; dead fast-path respawns immediately; concurrent callers share one respawn
  • spawner adoption: child exit(3) + live compatible twin → attaches to the twin; empty candidates → legacy failure text; foreign-lane candidate rejected; exit(1) path unchanged
  • eaddrinuseExitCode table; writeSpawnPortLog round-trip
  • refusal wording through the real createNativeRoute: saturated → "alive but unresponsive" (and asserted NOT to contain "guaranteed upstream 400"); under-window body → direct; dead proxy → legacy wording byte-identical

Mapping to the regression categories proposed in #3654: ① timeout-then-answer → covered (busy→alive, origin kept, request succeeds); ② truly dead → fail-closed with unchanged wording → covered; ③ load simulation → covered at contract level (bounded attempts, jitter present, no respawn inside the busy window) — a real 20 MiB compression + concurrent-probe simulation is deliberately outside the unit suite (the accept-queue physics are recorded in the liveness.ts header); ④ transport/call-site parity → both createLiveOriginResolver call sites ride the shared resolver; reactive lanes and the WS lane are noted above; ⑤ version continuity (folds surviving N→N+1) belongs to the #3573 family and is out of scope here.

Local targeted runs green: new suite 23/23 · native-intercept 63/63 · opencode-native 40/40 · launcher 326/326 · listen-policy 5/5 · issue3436-native-proxy-reuse 8/8 · port-owner 24/24 · port-race 4/4 · port-zone 17/17 · typecheck clean · build OK. Full-suite verdict = CI.

Config surface

None — no config fields, env vars, or CLI flags added.

@github-actions

github-actions Bot commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-18_health-probe-three-state (87c465e)

Option A — Install this exact build (recommended)

npm install -g billion-context@0.0.4386-pr.3673.4f28e41

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr3673.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei ranxianglei changed the title fix: three-state health probe — a busy proxy is no longer misjudged as dead (#3654) [PR] fix: three-state health probe — a busy proxy is no longer misjudged as dead (#3654) Oct 18, 2026
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重:不重复。 本 PR 是 #3654(探活把「忙」判成「死」→ 级联拒发)的唯一在途修复,Closes #3654。同族已核查:#3308/#3318(fail-closed 通道本身,本 PR 不动其语义)、#928/#930(探活 TTL,保留并扩展为三态缓存语义)、#3436/#3540(多实例层,已合入 cb02753,本 PR 基于其上)、#3116(0 字节诊断日志,本 PR 的 per-port 日志补行是其后续)。无并列重复 PR。

标签: bug / opencode / scope/wiring / p2 —— 缺陷类;主动探活路径仅 opencode native V2 + launcher 两条路由;协议接线层的存活判定与 spawn 收养;高负载下超窗会话被拒、低负载不受影响、有绕行 → p2。

这是 agent 已完成的 [PR](body 带 ework-agent-pr),我不重做工作,按规范独立审一遍:拉取分支核对代码、逐项回归清点、测试有效性、CI 状态,随后按八问给结论。分支已拉到本地(单 commit 9d5127762,base = 当前 master 1f0397b0a,无 stale-base 问题)。审查进行中,结论稍后补在这条线程里。

…#3654)

Review nit on 9d51277: the BUSY announcement line fired for every
concurrent caller entering the same episode while its comment claimed it
announces an episode ONCE, and the announcedAt field backing that claim
was never read. Gate the announcement on fresh-episode creation, drop
the dead field, and clear episode entries at the respawn generation
boundary (same place the verdict cache is rebuilt) so a post-respawn
busy episode behind the same origin announces itself instead of
inheriting a stale entry. Log-only; no behavior change.
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

独立审核完成:未发现阻断性缺陷,CI 绿即可合。 owner 在 #3654 要求的五个附加项逐条落实;回归审计逐项通过;测试文件是真测试。

八问:

  1. 是什么问题:高负载压缩把事件循环打满时,健康探测超时被当成"死了"→ 立即重生子进程 → 端口冲突 exit(1) → bootstrap 失败、超窗请求本地拒发;而原代理其实活着(curl 200)。
  2. 什么原因导致的:探测只有活/死两态,把"超时"归进了死。根因是事件循环饱和时 TCP accept 仍有内核应答、HTTP 却永远不应答——超时信号的真实含义是"忙"不是"死",单态混淆让系统把一个可恢复状态当终态处理。
  3. 影响范围与严重程度:仅 opencode native 通道(V1/V2),且需大压缩与请求并发才触发。p2:饱和窗口内用户遇到 bootstrap 失败和本地拒发,原实例外部仍可用;不影响 proxy 模式与小窗流量。
  4. 怎么修复的:三态判定(连接被拒=死 / 超时=忙 / HTTP 应答后按载荷判活死)+ 忙永不进 TTL 缓存 + held origin 有界等待窗(≤10s 抖动重探,复活即继续用原实例)+ spawn 侧收到 exit-3 收养已在服务的 twin(替代 exit(1) 硬失败)+ 拒发文案按原因分叉。
  5. 是否有回归问题:逐项审过——legacy dead 路径拒发文案被测试 F 字节级钉死(无漂移);claude-native hooks must not attach to a lifecycle-less proxy — attach gate + self-managed fallback #1335 发现层门控语义不变(busy 实例照丢,只改诊断文字);[Bug][OpenCode] 同 origin 代理重启复用旧负健康缓存,恢复首请求绕过压缩 #1957/fix(#1957): invalidate stale health verdicts across the respawn boundary #1966 代际失效纪律保留(respawn 重建缓存并作废在途判定);小窗请求 fail-open 直通不变;15s respawn 冷却不变(仍仅 ownedThenLost 绕过,与现状一致);watcher/心跳协议无语义变化。未见无意回归。
  6. 是否破坏缓存:不破坏。TTL 缓存现在只缓存确定判定(活/死),忙不缓存——这正是修复点之一。会话持久化格式未动。
  7. 是否改了设计方向:可用性契约从"探测失败=死"扩展为三态存活判定,即 [Bug?] Under high load, health check timeouts appear to be misinterpreted as "proxy unavailable": large session requests | [Bug?] 高负载下 health 探活超时疑似被当成「代理不可用」:大会话请求被本地拒发(0.1.199 / plugin native) #3654 floor 4 owner 拍板的方案 A;架构边界未动。
  8. 是否引入新配置:无。BUSY_WAIT_CAP_MS(10s)/BUSY_REPROBE_BASE_MS(1s) 是内部常量(测试可注入),不是 config/env 面。

行为变更清单(旧 → 新):

  • twin 冲突子进程退出码:1 → 3(无条件生效,含手动启动——手动场景没有收养者,3 只是"启动失败"的一种细分;strict-port/port-busy/upgrade-replacement 仍是 1)
  • 忙 origin 处置:立即丢弃+绕过冷却立即重生 → ≤10s 抖动重探窗口,复活则继续用原实例;耗尽才释放+重生(原因记为 busy-exhausted)
  • 重生出的替换实例处于忙态:原先无法识别(按死丢弃)→ 现在直接路由(请求排队等折叠完成)
  • 超窗拒发文案:统一 "proxy unavailable … guaranteed upstream 400" → 二分:busy-exhausted 用 "alive but unresponsive … retry shortly";dead/bootstrap-failed 保持原文案字节不动
  • 发现层丢弃诊断:"health probe failed" → 区分 BUSY(TCP 可达)与 dead
  • spawn 子进程额外向 per-port 日志写分类行(仅 launchToken 场景)

测试有效性:真测试——A 组真实 socket(paused socket 模拟饱和+耗时断言)、C 组经 fake 子进程 exit-3 端到端驱动 ensureProxyRunning(含 foreign-lane 不得收养的负例)、F 组新旧文案双向钉死。断言有牙,bug 回归必被抓。

顺手小修(独立标注 commit b5e02b0ee):BUSY 通告日志原本对同一 episode 的每个并发调用方各打一行,与注释声称的"每 episode 报一次"矛盾,且配套的 announcedAt 字段写了从未读。改为 episode 新建时才通告 + respawn 代际边界清理 episode 表。纯日志面改动、无行为影响;本地定向套件 + typecheck 绿。

CI:head 9d51277 尚无运行记录(bot PR 为 action_required,见 ci.yml #772)——需要你在 Actions 页面点 Approve-and-run;我推小修后 head 已是 b5e02b0ee,请对该 head 批准一次。本地定向套件 + typecheck 已过。

已知残留(低风险):收养一个正在饱和的 twin 时,首次 watcher 注册可能 2s 超时,退化为单 owner watchdog 语义;心跳 beat 每 5s 幂等重注册,≤~7s 自愈。不值得在本 PR 内加机制。

判定:可合,待 CI 绿。

…ate (#3654)

CI gate failure on the PR head: probeOriginHealth / LivenessState / CONNECT_PROBE_TIMEOUT_MS are used only inside their defining modules (drop the export); FailFastReason now has its intended external user - server.ts imports it instead of restating the union inline.
@ranxianglei
ranxianglei merged commit 9c44cfd into master Oct 19, 2026
110 of 154 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

📦 Merged Build Published

This merge to master was published as 0.0.583-master.9c44cfd (rolling master tag).

npm install -g billion-context@0.0.583-master.9c44cfd  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

HaqimIskandar pushed a commit to HaqimIskandar/billion-context that referenced this pull request Oct 27, 2026
…hem (ranxianglei#4811)

Discovery (probeLiveInstances) dropped a candidate whose health probe timed
out identically to one whose process was dead — even when TCP connect proved
the event loop merely saturated (ranxianglei#3654 three-state, discovery half deferred by
ranxianglei#3673). Consequence chain: discovery reports empty → spawn a second proxy on
the lane port → child opens the shared index.db before its listen fails →
EADDRINUSE → TWIN_ADOPT_EXIT_CODE detour (7-27s measured) racing the client's
hard ~13.5s give-up wall → lost ~half the time → session commits to direct
(uncompressed) traffic permanently. The spawned child also created a second
index.db writer during the window (ranxianglei#3923/ranxianglei#3959 family).

- probeLiveInstances: keep candidates that answer TCP but not health probes
  (health === undefined && connectable); an explicit ok:false response still
  means "answered, draining" and stays dropped; diagnostics distinguish the
  three outcomes
- pickAttachable: a BUSY candidate passes every health-free check verbatim
  (fingerprint, shape, strictPort, own-lane policy) and is attachable when its
  lifecycle is verifiable — own lane, attachExternal, or user-zone instance;
  a BUSY foreign declared lane is refused with a diagnostic instead of being
  silently dropped
- attachGateAllows / rotateStalePredecessor widened to HealthInfo | undefined
  (a BUSY instance can never be a rotation target: version unverifiable)
- isConnectable threaded through waitForStarterInstance / findLiveAttachable
  / rediscoverAfterChildExit call sites (deps override preserved)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

1 participant