Repository navigation
[PR] fix: three-state health probe — a busy proxy is no longer misjudged as dead (#3654) - #3673
Conversation
📦 Built Package ArtifactBranch: Option A — Install this exact build (recommended)npm install -g billion-context@0.0.4386-pr.3673.4f28e41This exact version is immutable and reproducible. The rolling Option B — Download artifact
tar xzf billion-context-pr3673.tgz
npm install -g packageInstall instructions are refreshed on each push. |
(ai: vllm-qwen/qwen3.8-27b) 查重:不重复。 本 PR 是 #3654(探活把「忙」判成「死」→ 级联拒发)的唯一在途修复, 标签: bug / opencode / scope/wiring / p2 —— 缺陷类;主动探活路径仅 opencode native V2 + launcher 两条路由;协议接线层的存活判定与 spawn 收养;高负载下超窗会话被拒、低负载不受影响、有绕行 → p2。 这是 agent 已完成的 [PR](body 带 |
…#3654) Review nit on 9d51277: the BUSY announcement line fired for every concurrent caller entering the same episode while its comment claimed it announces an episode ONCE, and the announcedAt field backing that claim was never read. Gate the announcement on fresh-episode creation, drop the dead field, and clear episode entries at the respawn generation boundary (same place the verdict cache is rebuilt) so a post-respawn busy episode behind the same origin announces itself instead of inheriting a stale entry. Log-only; no behavior change.
(ai: vllm-qwen/qwen3.8-27b) 独立审核完成:未发现阻断性缺陷,CI 绿即可合。 owner 在 #3654 要求的五个附加项逐条落实;回归审计逐项通过;测试文件是真测试。 八问:
行为变更清单(旧 → 新):
测试有效性:真测试——A 组真实 socket(paused socket 模拟饱和+耗时断言)、C 组经 fake 子进程 exit-3 端到端驱动 ensureProxyRunning(含 foreign-lane 不得收养的负例)、F 组新旧文案双向钉死。断言有牙,bug 回归必被抓。 顺手小修(独立标注 commit CI:head 9d51277 尚无运行记录(bot PR 为 action_required,见 ci.yml #772)——需要你在 Actions 页面点 Approve-and-run;我推小修后 head 已是 已知残留(低风险):收养一个正在饱和的 twin 时,首次 watcher 注册可能 2s 超时,退化为单 owner watchdog 语义;心跳 beat 每 5s 幂等重注册,≤~7s 自愈。不值得在本 PR 内加机制。 判定:可合,待 CI 绿。 |
…ate (#3654) CI gate failure on the PR head: probeOriginHealth / LivenessState / CONNECT_PROBE_TIMEOUT_MS are used only inside their defining modules (drop the export); FailFastReason now has its intended external user - server.ts imports it instead of restating the union inline.
📦 Merged Build PublishedThis merge to master was published as npm install -g billion-context@0.0.583-master.9c44cfd # this exact build
npm install -g billion-context@master # always the newest merged build
|
…hem (ranxianglei#4811) Discovery (probeLiveInstances) dropped a candidate whose health probe timed out identically to one whose process was dead — even when TCP connect proved the event loop merely saturated (ranxianglei#3654 three-state, discovery half deferred by ranxianglei#3673). Consequence chain: discovery reports empty → spawn a second proxy on the lane port → child opens the shared index.db before its listen fails → EADDRINUSE → TWIN_ADOPT_EXIT_CODE detour (7-27s measured) racing the client's hard ~13.5s give-up wall → lost ~half the time → session commits to direct (uncompressed) traffic permanently. The spawned child also created a second index.db writer during the window (ranxianglei#3923/ranxianglei#3959 family). - probeLiveInstances: keep candidates that answer TCP but not health probes (health === undefined && connectable); an explicit ok:false response still means "answered, draining" and stays dropped; diagnostics distinguish the three outcomes - pickAttachable: a BUSY candidate passes every health-free check verbatim (fingerprint, shape, strictPort, own-lane policy) and is attachable when its lifecycle is verifiable — own lane, attachExternal, or user-zone instance; a BUSY foreign declared lane is refused with a diagnostic instead of being silently dropped - attachGateAllows / rotateStalePredecessor widened to HealthInfo | undefined (a BUSY instance can never be a rotation target: version unverifiable) - isConnectable threaded through waitForStarterInstance / findLiveAttachable / rediscoverAfterChildExit call sites (deps override preserved)
Closes #3654
What changes
The health probe now distinguishes three states instead of two:
alive/__bili/healthanswered{ok:true}within budgetdeadbusyKey pieces:
src/liveness.ts(new) — sharedprobeLiveness(origin, budgetMs)+tcpConnectable(). A raw TCP-connect pre-probe is what makes busy/dead discriminable in milliseconds: the kernel accept queue answers connects while the JS event loop is blocked (measured: 2ms connect during an 8s block), while a refused port yieldsECONNREFUSEDimmediately.src/agent/native-intercept.ts—createLiveOriginResolver(both call sites: opencode native V2 route + opencode launcher route) consumes the verdicts: busy holds the origin without spawning (no cooldown bypass, no generation rebuild); the TTL cache stores alive/dead only, so busy is never cached as dead (per the [Bug][OpenCode] 同 origin 代理重启复用旧负健康缓存,恢复首请求绕过压缩 #1957/fix(#1957): invalidate stale health verdicts across the respawn boundary #1966 generation discipline); a replacement instance judged busy is routed to anyway (the request queues behind the fold); resolver failures recordlastUnavailableReasonfor downstream wording.src/launcher.ts,src/server.ts,src/listen-policy.ts,src/instance.ts) — a spawned child that loses the autoRestartOnUpdate + occupied target port -> orphan process leak: new instance silently hops to the next port, old instance never exits #3044 twin race now exits with code 3 (was 1) as an explicit adopt signal; the spawner verifies the serving instance (registry identity + live pid + TCP reachability) and attaches to it instead of failing the whole bootstrap. This removes the "discovery says dead / twin rule says alive" contradiction instead of de-frequencing it.$TMPDIR/bili-proxy-<port>.logthat was exactly 0 bytes in [Bug?] Under high load, health check timeouts appear to be misinterpreted as "proxy unavailable": large session requests | [Bug?] 高负载下 health 探活超时疑似被当成「代理不可用」:大会话请求被本地拒发(0.1.199 / plugin native) #3654); the discovery drop and the port diagnosis now report "BUSY (event loop saturated)" when TCP connect succeeds; every classification lands in bili.log with attempt counts.Behavior changes (old → new)
res.okonly{ok:true}JSON validationNo new config/env surface (option C was declined in the #3654 thread). Kernel untouched.
Deliberately unchanged (parity notes)
native-ws-intercept.ts) has no probe to misjudge —readyOriginonly; it benefits indirectly through the bootstrap adoption.shared.ts: 5s status timeout / 15s deadline) is unchanged — a boot misjudgment self-heals through spawn→adopt, which is now bounded by the same adoption path.recover()) keeps its shape (clear + respawn on an actual send failure); it benefits through bootstrap adoption.probeLiveInstancesstill drops candidates that fail the full attach gate (claude-native hooks must not attach to a lifecycle-less proxy — attach gate + self-managed fallback #1335) — now with a BUSY diagnostic attached.Tests
New
tests/issue3654-health-probe-three-state.test.ts(23 tests, hermetic, ephemeral ports vialisten(0)):probeLivenessagainst real sockets: alive / dead-fast (refusal classified <300ms) / busy (accepts, never answers) / foreign HTML 200 → dead /{ok:false}→ deadeaddrinuseExitCodetable;writeSpawnPortLoground-tripcreateNativeRoute: saturated → "alive but unresponsive" (and asserted NOT to contain "guaranteed upstream 400"); under-window body → direct; dead proxy → legacy wording byte-identicalMapping to the regression categories proposed in #3654: ① timeout-then-answer → covered (busy→alive, origin kept, request succeeds); ② truly dead → fail-closed with unchanged wording → covered; ③ load simulation → covered at contract level (bounded attempts, jitter present, no respawn inside the busy window) — a real 20 MiB compression + concurrent-probe simulation is deliberately outside the unit suite (the accept-queue physics are recorded in the
liveness.tsheader); ④ transport/call-site parity → bothcreateLiveOriginResolvercall sites ride the shared resolver; reactive lanes and the WS lane are noted above; ⑤ version continuity (folds surviving N→N+1) belongs to the #3573 family and is out of scope here.Local targeted runs green: new suite 23/23 · native-intercept 63/63 · opencode-native 40/40 · launcher 326/326 · listen-policy 5/5 · issue3436-native-proxy-reuse 8/8 · port-owner 24/24 · port-race 4/4 · port-zone 17/17 · typecheck clean · build OK. Full-suite verdict = CI.
Config surface
None — no config fields, env vars, or CLI flags added.