Skip to content

[PR] fix: surface failed watcher registration cause/state and report heartbeat recovery - #5644

Merged
ranxianglei merged 8 commits into
masterfrom
2026-11-04_watcher-registration-recovery
Nov 5, 2026
Merged

ranxianglei merged 8 commits into
masterfrom
2026-11-04_watcher-registration-recovery

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

Closes #5641

What was broken

When pi (or any plugin lane) attaches to an already-running shared proxy, it registers its own PID with the proxy's watchdog via POST /__bili/watcher so the proxy cannot exit while an owner is still alive. If that registration request fails transiently:

  1. the only trace was bili: watcher registration failed — … (fetch failed) — the underlying cause (connection refused, DNS failure, timeout, …) was lost;
  2. the returned handle kept only the 409 ("refused") outcome — a "failed" outcome was dropped, so nothing downstream could see that registration was never confirmed;
  3. the owner heartbeat (every 5 s) silently repaired the registration on the next accepted beat, but nobody ever reported the recovery — the alarming warning stood unexplained forever.

What changed (all in src/launcher.ts)

  1. Sanitized diagnostics (registerWatcherDefault + new exported formatWatcherError): the failure lines now carry the origin, the owner PID, the HTTP status when the request completed, and the first structured cause code from the error chain (falling back to the cause message for undici errors that expose no code, e.g. DNS lookups). A short tail states what happens next: own PID → "a later heartbeat will re-register us and report recovery"; foreign PID → "no heartbeat runs … stays unconfirmed unless a retry succeeds". No credentials, request bodies, or private endpoints anywhere in it.
  2. Registration outcome retained — ProxyHandle gains watcherRegistration: "ok" | "refused" | "failed", set on EVERY attach outcome; refusedWatcher is untouched for existing consumers. Attached/transport-ready no longer implies watchdog-acknowledged.
  3. Bounded retry for transient failures only — attach now makes up to 3 attempts with 500 ms backoff (worst case ≈ 7 s extra attach latency when the endpoint is down). 409 is never retried. This matters for foreign-PID callers, which intentionally run no heartbeat and would otherwise have zero recovery chance.
  4. Recovery acknowledged through the existing heartbeat channel — the worker inspects res.ok and posts a one-shot "ack" to the parent after the FIRST 2xx beat; the parent transitions pending→recovered exactly once and prints ONE host-visible notice through the existing diagnostic channel. A resolved fetch returning 409/5xx never counts as recovery. The in-thread fallback (worker spawn failure / mid-flight crash) gets equivalent semantics via the post() boolean result. No per-heartbeat logging, no watchdog death/expiry policy change, no spawn/restart/fallback added.

One Node runtime gotcha handled along the way: attaching a message listener to the (unref'd) heartbeat worker RE-REFS its MessagePort and would pin the host process open — unref() is re-asserted at attachment and after every delivery, preserving the #5152 liveness contract (verified with dedicated process-exit probes).

Behavior changes (disclosed)

Surface Before After
Transport-failure log line bili: watcher registration failed — the shared proxy may exit when its first owner does (fetch failed) same core phrase + origin + owner pid + cause detail + outcome-specific tail
HTTP-failure log line … returned HTTP <status> — … (no origin/pid) + origin + owner pid + tail
Attach latency while /__bili/watcher is down 1 attempt (≈2 s timeout) up to 3 attempts (≈7 s worst case), then attaches anyway
ProxyHandle refusedWatcher set only for 409 + watcherRegistration set for all outcomes (additive field)
Heartbeat worker protocol silent beats one-shot "ack" parent message after first 2xx beat (worker-internal only; wire shape {pid, hb:true} unchanged)
New log output — one recovery notice per recovered pending failure: bili: watcher registration recovered at <origin> (owner pid <pid>) — …

No change to watchdog arming/expiry, 409 semantics, or the external-attach policy.

Config surface

N/A — no new env vars, config keys, or CLI flags.

Tests

New tests/issue5641-watcher-registration-recovery.test.ts (14 tests) covers the issue's six-row regression table: transport failure → later 2xx beat → single recovery notice + zero spawns; persistent failure → bounded retries, stays unconfirmed; 409 → refused semantics preserved, never retried; 500/409 heartbeat responses do NOT count as recovery; foreign PID → no heartbeat started, outcome retained; worker crash → in-thread fallback with the same ack semantics. Plus an end-to-end attach test driving the real 5 s-interval worker.

Targeted local runs green: this suite (14/14, clean process exit), watcher-registration, issue5152-owner-beat-survives-host-stall (6/6), issue3544-stale-proxy-rotation (20/20), launcher (345 pass / 1 pre-existing skip); typecheck clean. Full-suite verdict: CI on this head.

ework-agent and others added 4 commits November 4, 2026 02:07
…4599)

Master 9dc4c94 red on test (windows-latest, 22, 2) only: the two SSE
abort cells exhausted the 10s teardown poll (closed=0, ~11s each) while
the same code went green on ubuntu22/24 and windows24 shard-2 siblings
(pass durations 23-352ms).

The audit mandated by the constant's own note found no teardown hang:
relay.ts arms res.on('close') -> clientAbort.abort() synchronously with
no linger/backstop on this path, and no commit since d244fff touched
destroy/abort wiring. Residual latency is runner-side event-loop
starvation on loaded Windows runners (same family as the recorded 3s/6s
eras). Platform-scoped response: win32 doubles to 20s; every other
platform keeps the tight 10s gate so a genuine hang still fails fast.
…beat recovery (#5641)

A transient POST /__bili/watcher failure on attach printed a bare "fetch failed" line, dropped the outcome from the returned handle (only 409 was retained), and then self-repaired silently via the owner heartbeat with no trace.

- registerWatcherDefault diagnostics carry sanitized origin + owner pid, the error name/message plus the first structured cause code (or cause message for undici errors that carry no code), and an outcome-specific tail (own pid: a later heartbeat re-registers and reports recovery; foreign pid: stays unconfirmed unless a retry succeeds).

- ProxyHandle gains watcherRegistration ("ok"|"refused"|"failed") for every attach outcome; refusedWatcher is kept for existing consumers.

- attachTo retries transient failures only (never 409): up to 3 attempts, 500ms backoff, worst-case ~7s extra attach latency.

- The heartbeat worker inspects res.ok and posts a one-shot "ack" after the first 2xx beat; the parent transitions pending->recovered exactly once through the host-safe diagnostic channel. The in-thread fallback path acks via the post() boolean result. A resolved fetch returning 409/5xx never counts as recovery. Attaching the "message" listener re-refs the worker port (Node MessagePort semantics), so unref is re-asserted at attachment and after every delivery to keep the #5152 liveness contract.

No watchdog death/expiry policy change; no spawn/restart/fallback added; no config surface change.

Tests: tests/issue5641-watcher-registration-recovery.test.ts covers the issue six-row regression table.
@github-actions

github-actions Bot commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-11-04_watcher-registration-recovery (50318a4)

Latest build — install this exact version

npm install -g billion-context@0.0.8100-pr.5644.6e63e8f

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Recent builds (commit → version)

commit version build
50318a4 billion-context@0.0.8100-pr.5644.6e63e8f build
30b18d1 billion-context@0.0.8096-pr.5644.121f7b8 build
71c2675 billion-context@0.0.8093-pr.5644.ef78dee build
c8270e7 billion-context@0.0.8090-pr.5644.3c33f10 build

Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr5644.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei ranxianglei changed the title fix: surface failed watcher registration cause/state and report heartbeat recovery [PR] fix: surface failed watcher registration cause/state and report heartbeat recovery Nov 4, 2026
ework-agent and others added 2 commits November 4, 2026 13:03
#5428 made inline the pi lane default; this suite documents the spawn
contract (instance record + proxy origin, per-request stamps, the
shared-proxy delegate-child channel, /__bili/plugin/status over HTTP)
and was not updated with the flip. Under the inline default its
delegate-child stamp assertions get 0 stamped child requests (child
traffic reaches upstream unstamped, conv=null) and the /acp instance-
record assertion fails structurally. Pin BILI_PI_INLINE=0 for the whole
suite so the spawn-posture coverage stays deterministic; inline posture
is covered by tests/pi-inline-ws.test.ts / tests/pi-native.test.ts.
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重: 不重复 — 无其他开放 issue/PR 处理 watcher 注册诊断/恢复;唯一相邻的 #5625 是 win22-s2 的 CI 抖动修复,与本 PR 无关。

Review of PR #5644 — 独立审核结论

方向关(①):原 issue #5641 报三个缺陷——失败丢失底层原因、handle 丢弃 "failed" 结果、心跳补登记后无恢复信号。PR 三项全中,外加有界重试(理由成立:foreign-pid 调用方如 claude 车道没有心跳兜底,重试是其唯一修复路径;own-pid 则缩短未登记窗口)。修在决策点(注册结果的处理与上报),不是症状掩盖。版本核对:报告方 v0.1.212(daa082e),master head 23a826d 同样声明 0.1.212 且缺陷机制在 base 上确认存在。✓

回归审查(②)— 触碰路径的既有行为清单,逐项判定:

既有行为 结论
watchdog arming/expiry 语义 保持(server 侧零改动)
409/refused 语义(#1322) 保持:仍返回 "refused"、置 refusedWatcher、注册层静默、从不重试(测试钉死 calls===1)
外部 attach 策略(#1335/#2905/#2877) 保持(零触碰)
心跳 wire 形状 {pid,hb:true} 保持(e2e 测试 deepEqual 钉住)
#5152 unref/liveness 契约 保持且正确:message listener 会 re-ref MessagePort,代码在所有监听器挂完后 unref + 每次投递后再 unref;issue5152 套件本地 6/6 绿
#3544 注入 post 缝 兼容:result === true 门控,旧调用方(resolves undefined)行为不变;src/ 内无任何生产调用方传 post;套件本地 20/20 绿
attach 时延(ok/refused 路径) 保持:仅 "failed" 才进重试环,ok/refused 一次即出(calls===1 双向钉死)
ProxyHandle 消费方(claude-mcp-entry.ts:90、claude-native-bootstrap.ts:506 读 refusedWatcher) 不受影响(纯增量可选字段)
spawn/starter-poll 各 startOwnerHeartbeat 调用点 签名未变,行为同 attach 路径分析

无无意回归。高危模式逐项过:无 fail-fast→吞错切换(注册失败仍降级不阻塞 attach,与旧行为一致)、无阈值/时序漂移超出披露范围、无输出格式变化超出披露范围、无缓存/去重策略触碰。

代码审查(③):逻辑干净。formatWatcherError 限深 3、只取 name/message/code,不含 body/凭据;DNS 类无结构化 code 的故障走 cause message 回退(Node 22 实测形态)。恢复通知走 attach 时捕获的 attachDiag 宿主安全通道(#3404),不在 attach 路径引入裸 console.*。watcherPendingRecovery 三条清除路径齐全(ack 恢复 / 直接 ok / stop())。

测试有效性(④):真测试。断言有牙且双向:精确计数(calls===3、sleeps=[500,500])、精确日志子串、wire 形状 deepEqual、负向行齐备(500/409 应答不算恢复、falsy post 不算 ack、refused/ok 都不重试)、幂等(恰好一条通知、settled 后无刷屏)、stop() 冻结 fallback 心跳、e2e 走真 ensureProxyRunning + 生产 5s 间隔 worker。临时端口 listen(0),无端口假设。这个 bug 回归时抓得住。

本地验证(定向,非全量):rebased head 上 tsc --noEmit 干净;四个相关套件 41/41 绿(issue5641 14/14、watcher-registration、issue5152 6/6、issue3544 20/20)。

CI 状态与处置:c8270e75c 上 47 项检查,唯一红 = test (windows-latest, 22, 2):tests/wire-exit-matrix.test.ts:234 "abort x proxy-openai-sse: client abort mid-stream destroys the upstream request"(closed=0,10.6s 超时断言)——socket teardown 时序抖动,继承性 master 红,与本 PR 无关(diff 只碰 launcher.ts,该测试路径零交集;#5625 已记录同一 flake,ci-fixes 哨兵上已有对应修复 9d1eb15)。已按 CI 协调规程把 origin/ci-fixes merge 进 PR 分支并推送,新 head sha=71c267515,CI 重跑中。(注:#4599 总控线程在本沙箱检索不到内部编号,状态行在此代发:PR #5644 红为继承性,已 merge ci-fixes,sha=71c267515。)

小观察(不拦合并,同一通知精度家族):多 attach 拓扑下恢复通知的归属可能偏移——① foreign-pid(claude 车道)注册失败也会 mark pending,但 foreign pid 不起心跳、stop() 永不到来,该条目驻留至进程退出(几字节×origin 数,无功能影响);若同进程另一车道随后以 own-pid 对同一 origin 成功 attach,其 beat 的 ack 会清掉这条 pending 并发出「recovered (owner pid )」通知,而服务端实际重加的是 beat 方的 pid——noteWatcherHeartbeatAck 不比对 pid。② 同 origin 的 worker 若在第二次 attach 失败前已消耗一次性 ack,后续 2xx beat 不再补发通知(功能恢复仍由 beat 完成,仅缺一条日志)。两者都要求「同进程、同 origin、两次 attach」的罕见拓扑,最坏后果是一条归属不准/缺失的通知。建议(留给你定):ack 携带并比对 pid,或 startOwnerHeartbeat 返回 undefined(foreign pid)时不 mark pending。

行为变更披露核对(Deliverables 第 4 条):PR 描述的六项变更逐一与实现比对,全部属实,无隐藏漂移——失败日志行扩写(origin+pid+cause+tail)、endpoint 不通时 attach 时延 2s→≤7s、ProxyHandle 增量字段、worker 内部一次性 "ack"、每 episode 一条恢复通知。配置面:不涉及。


结构化八问

  1. 是什么问题:pi/plugin 车道附着共享代理时,owner-watcher 注册的临时失败只剩一句 "fetch failed"——丢原因、丢状态(handle 只记 refused)、后来心跳悄悄修好了却没有任何信号,用户面对一条永远无法从日志判断现状的生命周期警告。
  2. 什么原因导致的:注册结果处理层只建模了两种结局(ok/refused),传输故障被折叠成无信息的 "failed" 且不入 handle;诊断行未展开 nested cause;心跳发送端把「beat 被接受(=服务端重加 pid)」当作纯副作用,从不回报——三个环节各自合理,合起来就是不可观测的状态黑洞。
  3. 影响范围与严重程度:所有附着共享代理的车道(pi native、claude、omp 等),仅在注册请求撞上瞬时网络故障时触发(事件循环饱和期的 TCP reset/超时,见 issue 现场);严重度 p2——流量照常、多数情况心跳自愈,但用户无法判断代理是否会在首个 owner 退出时带走全部会话,且故障本身难诊断。非数据丢失(issue 现场确认已自愈、无数据损坏)。
  4. 怎么修复的:在决策点补齐三件事——失败诊断展开到第一个带 code/message 的 cause(脱敏,只取 name/code/message);handle 携带全部三种结局,attached 不再隐含 registered;首次被确认的 beat(2xx)驱动 pending→recovered 一次性通知,worker 主路径与线程内 fallback 同语义。另加有界重试(3 次/500ms/≤~7s),因为 foreign-pid 调用方没有心跳兜底。
  5. 是否有回归问题:无。上表九项既有行为逐项过:watchdog 语义、409 语义、外部 attach 策略、wire 形状、[Bug] Watchdog takes its stale-heartbeat reading on its own event loop, so a synchronous stall kills a live proxy (parent-gone → uncompressed 400) #5152 liveness、安装新版本后旧代理永远不换新:detach 代理躲过应用退出,重启只是重新挂上旧进程(「永远更新不了」) #3544 缝、ok/refused 时延、handle 消费方、spawn 路径全部保持;无 fail-fast 变吞错、无阈值漂移、无格式漂移。
  6. 是否破坏缓存:不涉及——不碰前缀缓存、压缩、会话存储、重试游标任何一面。
  7. 是否改了设计方向:不涉及架构转向。是在既有 kimi native 共享代理存在与 #1183 同类的「首 owner 退出杀共享代理」暴露面(无 watcher 注册) #1190/[bug] claude native proxy outlives all sessions despite README lifecycle contract — watcher-based shutdown did not fire #1322/安装新版本后旧代理永远不换新:detach 代理躲过应用退出,重启只是重新挂上旧进程(「永远更新不了」) #3544/[Bug] Watchdog takes its stale-heartbeat reading on its own event loop, so a synchronous stall kills a live proxy (parent-gone → uncompressed 400) #5152 机制上补齐可观测性;新增的模块级 pending map 与两个导出函数是该方向的直接延伸。
  8. 是否引入了新的配置:无新 env/config/默认值。重试预算是模块内常量(WATCHER_REG_ATTEMPTS=3、backoff 500ms),非用户面旋钮。

判定:可合并 —— 代码与测试质量达标,行为变更全部如实披露;唯一待办是新 head 71c2675 的 CI 最终绿(预期:继承性 win22-s2 flake 已被 ci-fixes 上的预算修复覆盖)。上述小观察建议作为后续精修或并入你已有的车道计划,不阻塞本 PR。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

CI 终态确认:新 head 71c267515(merge ci-fixes 后)49 项检查全部完成——success/skipped,零失败;此前唯一的继承性红 test (windows-latest, 22, 2)(wire-exit-matrix abort teardown 抖动)已被 ci-fixes 上的预算修复覆盖,不再复现。

审查结论维持:可合并,无遗留阻塞项。合并后若处理那条小观察(pending 条目在 foreign-pid/多 attach 拓扑下的通知精度),按「同一机制一个 issue」归入 watcher 注册家族即可。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

七问(本次 auto-fix run,attempt 1/3)

  1. 是什么问题:PR 的 CI 只有 test (windows-latest, 22, 2) 红。该 shard 2553 个测试恰挂 1 个:tests/wire-exit-matrix.test.ts 的 abort x proxy-openai-sse: client abort mid-stream destroys the upstream request,断言 upstream socket(s) must be destroyed after the client abort (closed=0),耗时 10672ms——正好打满 10s teardown 轮询预算。
  2. 什么原因导致的:不是本 PR 代码回归。本 PR 只改 src/launcher.ts watcher 注册/心跳 attach 路径 + 新增测试文件;失败断言覆盖的是 relay teardown(客户端 abort 后销毁上游 socket),与 diff 零交集。根因是高负载 Windows runner 上的事件循环饥饿([PR] release v0.1.209 #4982/[PR] fix: split header-less claude code subagents onto their own session lane (#4352) #4774/[PR] fix(window): size a child conversation from its parent's runtime-info #4683/Flaky Windows CI: issue1993-preflight-400 suite cancelled by early event-loop drain | Windows CI 偶发假红:preflight-400 测试进程提前退出致 5 例取消 #3665/windows-22-1 lane randomly red: during test cleanup, unlink index.db hit an EBUSY external lock, couldn't survive tmp-rm | windows-22-1 车道随机红:测试清理期 unlink index.db 遇 EBUSY 外部锁,熬不过 tmp-rm 的 9.6 秒重试梯子(#2799/#2908 族复发) #4347 同族,历来只现于 windows-latest,ubuntu 从未红):审计证据显示 relay.ts 同步 arm close→abort、无 linger 路径;同 run 里兄弟 cell anthropic-sse 335ms 通过、其余含 windows-22 全部其他 shard 全绿。签名与 master 9dc4c94(2026-11-03,CI Control Center: Centralized CI Failure Handling & Status Sync (Comment all CI issues here; do not open new issues) | CI 总控: CI 失败集中处理与状态同步(所有 CI 问题在此评论, 勿开新 issue) #4599 floor 889 记录)的继承红完全一致。
  3. 影响范围与严重程度:仅该 PR 的 CI 判定被这一个 shard 挡下,无任何用户面/产品影响。严重度:CI 调度阻塞级。
  4. 怎么修复的:按 CI Control Center: Centralized CI Failure Handling & Status Sync (Comment all CI issues here; do not open new issues) | CI 总控: CI 失败集中处理与状态同步(所有 CI 问题在此评论, 勿开新 issue) #4599 常驻收敛纪律,继承红不在自己分支里排查改码,直接 git merge origin/ci-fixes(其中 9d1eb15 即针对此签名的 tests-only 修复:win32 的 ABORT_TEARDOWN_BUDGET_MS 10s→20s,其他平台保留 10s 紧门让真 hang 仍快速失败)。实际推送时发现并列 ci-check worker 已于 05:17:44Z 把同一 merge 推上分支(head 71c2675);我校验了自己独立产出的合并树(a3fc53ba5)与远端逐字节一致,故不重复推送等值提交。本地验证已过:typecheck clean + 合并树上 wire-exit-matrix 7/7 绿(Linux 上预算保持 10s,非 win32 行为未变)。
  5. 是否有回归问题:无。进入分支的净 diff 仅 3 个测试文件(+34/-6):wire-exit-matrix win32 预算三元式 + e2e-pi-codex-ws/e2e-native-pi 场景钉回 pre-kernel authorization request: release v0.1.211 bundles kernel 0.0.176 (#5415 abort-absorb nudge debt) #5428 姿态(BILI_PI_INLINE=0)。产品代码零改动;高危模式逐项过:无默认值/阈值语义漂移(仅 win32 测试预算放宽,且是哨兵审计后的平台化收敛)、无吞错、无输出格式变化。
  6. 是否破坏缓存:不涉及(无存储/缓存/会话相关改动)。
  7. 是否引入新配置:不涉及(无新 env/config/默认值)。

收工判定:head 71c2675 上 30 项 check 全部 completed、零失败(原红的 win22-s2 已转绿),本次红 CI 修复完成,PR 待人工合并。另在 #4599 留了一行收敛记录。

…ies (#5641)

Review nit (comment-only): the doc comment listed three ways a pending-recovery entry dies, but a foreign-owner-pid entry has no heartbeat to stop it and lives until process exit or a later direct-ok attach to the same origin. State that so the residual is visible in-code. No behavior change.

@ranxianglei ranxianglei left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(ai: vllm-qwen/qwen3.8-27b)

APPROVE(comment-review 形式提交——GitHub 禁止提交者正式 approve 自己的 PR,结论不受影响)。 独立全量审阅(diff 逐 hunk 对 master 23a826d37 = 当前 base 通读,不依赖 issue 侧结论与线程内先前审核);CI 在 head 71c267515 全绿(镜像缓存:30 checks = 17 success + 13 skipped,0 failed/pending)。

结构化八问

  1. 是什么问题:attach 到已在运行的共享 proxy 时,watchdog 注册请求(POST /__bili/watcher)瞬时失败后——日志只打 fetch failed 看不到真实原因、返回的 handle 丢掉这次失败结果(只有 409 有标志)、后续心跳实际悄悄修好了注册却没有任何可见信号,警告永久悬着与实际状态脱节。
  2. 什么原因导致的:链路上三处信息丢弃点——① registerWatcherDefault 只打 err.message,undici 的嵌套 cause(ECONNREFUSED/DNS 等)与 origin/pid 上下文丢失;② attachTo 只为 409 写 refusedWatcher,"failed" 结果无处落地;③ 心跳 worker/fallback 发送端从不看 HTTP 状态码,服务端在接受的 beat 上重新登记 PID 后没有通道回报父进程。修复对准三个丢弃点本身,不是治标。
  3. 影响范围与严重程度:所有走 attach 路径的插件车道(pi/opencode/dsh/claude 等);仅可观测性降级——心跳 ~5s 内自愈,无数据丢失、无可用性损失(p2)。
  4. 怎么修复的:四个机制全部在 src/launcher.ts——① 脱敏诊断(origin + owner pid + cause 链首个结构化 code,无 code 回退 cause message;仅 name/message/code,不落 body/凭据);② ProxyHandle.watcherRegistration: ok|refused|failed 每次 attach 结果都落字段(旧 refusedWatcher 原样保留);③ 只对瞬时失败做有界重试(3 次、500ms backoff、409 永不重试)——这是不启动心跳的外部 PID 调用方唯一的补救机会;④ 恢复确认复用既有 worker/parent 通道:worker 检查 res.ok,首个 2xx beat 后发一次性 "ack",父进程 pending→recovered 恰好一次、经 host-safe 诊断通道输出;409/5xx 应答永不算恢复;in-thread fallback 同等语义(post 严格 resolve true 才计)。独立核实:三条 attach 路线(waitForOtherStarter/adoptServingTwin/rediscoverAfterChildExit)全部收敛进 attachTo,无绕过;spawn 路径不需要注册(子进程原生跟踪 BILI_PARENT_PID),不受影响。
  5. 是否有回归问题:未发现。逐项清点既有行为:wire 形状 {pid}/{pid,hb:true} 不变(测试逐字钉住);409→refused 语义与 claude-mcp-entry/claude-native-bootstrap 消费链不变;registerWatcherDefault「永不抛」契约不变;#5152 worker unref 活性契约保留并加固(挂 message 监听器会 re-ref port——监听器注册完成后及每次投递后重新 unref,顺序核实正确);#3544 injected-post seam 保留(既有测试的 post resolve undefined 不会被误判为 ack);OwnerBeatWorker 接口扩宽向后兼容(CI typecheck 全绿);spawn 路径 5 处 startOwnerHeartbeat 调用点为纯上下文行未动;kernel/ 未触、config 面未动、version 字段未动。行为变更披露(均为增量):两条失败日志行内容变丰富(核心句 "the shared proxy may exit when its first owner does" 逐字保留,旧 grep 仍命中);/__bili/watcher 不可用时 attach 最坏比之前多花 ~7s(之前单次尝试即附接)——有界、仅失败路径发生、正常路径零开销;每次注册失败事件多一条恢复通知(至多一条)。
  6. 是否破坏缓存:不涉及(无缓存/重试游标/去重键变更)。
  7. 是否改了设计方向:不涉及——完全留在既有 worker-based heartbeat 设计内,未新增任何 spawn/restart/fallback,watchdog 策略不变。
  8. 是否引入新的配置:不涉及(无新 env/config/CLI 字段)。

测试有效性

真测试,断言有牙。14 个用例把 issue 场景表六行全覆盖且正反方向齐备(gate 双向:409/500 beat 不算恢复、falsy post 结果不算 ack、干净注册不重试、409 永不重试);用真实 HTTP stub(listen(0) 临时端口)+ 真实 Worker(非仅 fake);端到端用例驱动生产 5s 间隔的 worker 走真实 ensureProxyRunning attach 路径,spawnImpl 直接 throw 钉死「不得多开 proxy」;确定性 seam(deps.sleep/registerWatcher/injected post)保证 CI 可复现。四个机制任何一个回归都会让对应行变红。

Diff 整洁度

主提交只触 src/launcher.ts + 新测试文件;其余为 merge ci-fixes 带入的测试调整(e2e pi 套件钉 pre-#5428 spawn posture、win32 teardown 预算 10s→20s 平台限定),均带 provenance 注释、#4599 哨兵收敛、已逐一目检,test-only。无无关改动、无空白噪音、无 lockfile 漂移。

已知残留(非阻塞,记录在案)

「多车道共享同一宿主进程」的罕见拓扑下:外部 PID attach 失败留下 pending entry 后,若原 own-pid worker 中途死亡,fallback 循环的首个 2xx beat 可能发出一条标注外部 pid 的恢复通知(实际被重新 attest 的是 process.pid)——触发需三重巧合叠加,后果是一条误导日志行,功能性 attest 不受影响;另同一 origin 同进程内的第二次 attach 失败(worker 一次性 ack 已消费)不会再有通知(功能上仍由 beat 自愈)。两者属新功能的设计取舍,代码不动,留给作者后续判断。

已顺手修:watcherPendingRecovery 文档注释未说明 foreign-pid entry 的生命周期(无心跳可停它,活到进程退出或同 origin 下一次 direct-ok attach)+ commit 30b18d156(comment-only,无行为变更)。

判定:可合入。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

Review 完成,结论:可合入(APPROVE,comment-review 形式提交——GitHub 禁止提交者正式 approve 自己的 PR)。

…5644)

win22-s2 red on head 30b18d1: 'a delta re-claiming a live canonical id'
read ['u1'] where ['a1','u1','u2'] was expected — the HTTP turn flow was
correct (200/400/appliedSeq all asserted fine); only the durable rows were
missing. Mechanism: the lane wrap-up arms a debounced save AFTER the ack
(incremental-lane.ts), whose async write chain fsyncs behind the test's
fixed 60ms settle sleep; under loaded windows-latest runners the fsync
outran the budget and rows() saw the pre-turn snapshot. Same code green on
the previous head (71c2675) and on every other platform/shard.

Replace the fixed sleep with the store's own landing verdict
(afterNextFlush, #5155/#4788 D2 — the same signal the production pointer
upkeep already awaits): deterministic when the save is slow, and a false
verdict fails loud at the verdict instead of as a confusing missing-row
diff if a save genuinely fails.
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

七问(本次 auto-fix run,attempt 1/3)

  1. 是什么问题:head 30b18d1 只有 test (windows-latest, 22, 2) 红。挂的是 tests/msgid-lane-flow.test.ts 的「a delta re-claiming a live canonical id hard-fails (400), fresh ids append」:从 SQLite 读出行 id 为 ['u1'],期望 ['a1','u1','u2']——测试里所有 HTTP 层断言(seed 200、poison delta 400+reuse、good delta 200+appliedSeq=3)全部通过,只缺最后一轮新写的两行。
  2. 什么原因导致的:不是本 PR 代码回归。本 PR diff 只动 src/launcher.ts watcher 注册路径(+4 行注释),而这个测试走进程内 startServer,根本不经过 launcher。根因是测试 harness 的竞态:lane 的 wrap-up 在响应 ack 之后才 arm 15ms 防抖保存(src/incremental-lane.ts:892 forward(res) → :910 scheduleSave),保存写链含异步 fsync;高负载 Windows runner 上 fsync 超过了测试里固定的 60ms settle sleep,rows() 打开 DB 读到的是转前快照。非确定性证据:前一 head 71c2675(同代码少 4 行注释)win22-s2 全绿;同 run 其余 shard(含其他 windows-22)全绿;本地 Linux 修复前 5/5 绿。
  3. 影响范围与严重程度:仅本 PR 的 CI 判定被一个 shard 挡下,无任何用户面影响——存储层行为未变(生产侧指针维护本来就 await 同一个落盘信号,失败的防抖写由下一轮 re-arm 覆盖,崩溃窗口按既有 409-resync 契约兜底)。严重度:CI 调度阻塞级。
  4. 怎么修复的:tests-only 小修——harness 的 settle() 从固定 60ms sleep 改为 await store 自己的落盘判决 afterNextFlush("conv-msgid-1")(The incremental lane re-renders the entire conversation history and synchronously flushes it to disk each turn; large co | 增量车道每轮把整段会话历史重新渲染并同步落盘,大对话每轮多花约100毫秒 #5155/[Performance] v0.1.208 long sessions exceed 2 GiB memory and spend 14–22 seconds in local processing #4788 D2 API,与生产指针维护 await 的是同一信号):写慢就等到真正落盘再读行;写真失败则在判决处大声断言失败(fail loud),而不是留下一个误导性的缺行 diff。本地验证:该文件 6/6 × 4 次全跑 + typecheck clean。commit 50318a49b 已推分支,重跑后原红 win22-s2 转绿、全矩阵零红。
  5. 是否有回归问题:无。净 diff 仅该测试文件 +14/-4,产品代码零改动;debounceMs=15 等 harness 参数未动,无默认值/阈值/时序/输出格式漂移;断言强度只增不减(verdict=false 时显式失败),无吞错方向的变化。
  6. 是否破坏缓存:不涉及(无存储格式/缓存相关改动)。
  7. 是否引入新配置:不涉及(无新 env/config/默认值)。

收工判定:head 50318a4 上 30 项 check 全部 completed、零失败,本次红 CI 修复完成,PR 待人工合并。另在 #4599 留了备注:三个兄弟测试文件(seq-interval-lane / incremental-lane / issue5155-incremental-per-turn)共享同一固定 sleep settle 模式,当前无红,收敛与否留给哨兵裁量。

@ranxianglei ranxianglei left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(ai: vllm-qwen/qwen3.8-27b)

APPROVE(第 2 轮增量复审,comment-review 形式——GitHub 禁止提交者自批)。 本轮 delta = 我上轮审阅状态 30b18d156 之上新增的唯一一个提交 50318a49b(test-only:tests/msgid-lane-flow.test.ts +14/-4);CI 在新 head 50318a49b 全绿(镜像缓存:17 success + 13 skipped,0 failed/pending)。上一轮对主修复(launcher.ts 四机制)的完整八问结论继续有效,本轮只审增量。

增量独立核实(不依赖提交信息自述)

  1. 根因合理性:win22-s2 在 30b18d156 红于 'a delta re-claiming a live canonical id'(rows 读到 pre-turn 快照)。该测试与本 PR 的 launcher 改动无任何耦合——机制是 #5155 的 debounce 落盘(15ms)+ fsync 在负载型 windows-latest runner 上跑赢了测试里写死的 60ms settle sleep。证据链自洽:同一代码在上一个 head 与其余全部平台/shard 均绿。这是典型的「sleep 预算 vs 完成信号」竞态,修法方向正确。
  2. 新 settle 语义逐行核实(src/persist.ts:3064 afterNextFlush):无 pending flush 时立即 resolve true(incremental-lane.ts:916 注释明示,生产 pointer upkeep 已依赖此语义)→ 不会挂死;save 真失败时 waiter 以失败判决 drain → 测试 assert.ok(ok) 在判决点 loud-fail,而不是留下令人困惑的 missing-row diff。严格比旧行为更强(旧 settle 恒真),且给测试加了牙。
  3. 会话键核对:afterNextFlush("conv-msgid-1") 与测试实际使用的 x-bili-plugin-conversation: conv-msgid-1 / SQL 查询键一致,无错键等待。
  4. 无重复修复:origin/master 与 origin/ci-fixes 上该文件历史一致(最新 f13707f),均不含此改动——本分支独有,合入后随 master 收敛;按几行级小修政策,CI 红小修以标注 commit 落在当前 PR 是允许的形态。
  5. diff 整洁度:单文件、纯测试、注释同步更新、无空白噪音。

结构化八问(本轮增量范围)

  1. 是什么问题:msgid-lane-flow 测试用固定 60ms sleep 等 debounce 落盘,负载型 Windows runner 上 fsync 跑赢预算导致读到旧快照(CI 红,与本 PR 功能无关)。
  2. 什么原因导致的:测试确定性依赖时间预算而非完成信号(§7.2「no environmental luck」的反例)。
  3. 影响范围与严重程度:仅 CI 稳定性(win22-s2 偶发红),不影响产品行为。
  4. 怎么修复的:settle 改为 await 存储自身的落盘判决 afterNextFlush(sessionId) 并断言其真值——慢则等、真失败则在判决点响亮失败。
  5. 是否有回归问题:无。产品代码零改动;测试语义只强不弱(恒真 sleep → 带断言的完成信号)。
  6. 是否破坏缓存:不涉及。
  7. 是否改了设计方向:不涉及。
  8. 是否引入新的配置:不涉及。

判定:可合入。 两轮合计:主修复四机制成立、回归清点干净、测试有牙、diff 整洁;本轮增量是一处正确的测试确定性修复,无新增风险。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

第 2 轮增量复审完成(head 50318a49b),结论维持:可合入(APPROVE,comment-review 形式——GitHub 禁止提交者自批)。

@ranxianglei
ranxianglei merged commit e4c50ef into master Nov 5, 2026
49 checks passed
@github-actions

github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

📦 Merged Build Published

This merge to master was published as 0.0.1035-master.e4c50ef (rolling master tag).

npm install -g billion-context@0.0.1035-master.e4c50ef  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

Louie0806 added a commit to Louie0806/billion-context that referenced this pull request Nov 5, 2026
…mpress gate, ranxianglei#5686 dsh drift consolidation, ranxianglei#5696 model-scoped keyring 429

Ride-along upstream work that landed after c9ba485:
- ranxianglei#5683 forbid unprompted / out-of-bounds compression
- ranxianglei#5686 dsh drift consolidation (last native-compaction leak window)
- ranxianglei#5696 model-scoped KeyRing rate-limit holds
- ranxianglei#5661 incremental-lane credential headers, ranxianglei#5656 single-shot sidecar
- ranxianglei#5644 watcher registration recovery, ranxianglei#5635 restart survival

Overlap with this lane is limited to regenerated artifacts (CONFIGURATION*.md)
and to the shared relay/core seams, both auto-merged. No source conflict.

Verified after rebuilding kernel: issue5175 58/0, tag-echo 79/0,
issue2190-degen-close 32/0, golden-storage-render 5/0,
storage-engine-proofs 12/0, native-compact-terminal 1/0,
plugin-passthrough-tag-strip-chat 20/0, plus the newly arrived
ranxianglei#5683/ranxianglei#5686/ranxianglei#5656/ranxianglei#5658/ranxianglei#5631/ranxianglei#5641 suites all green.
tsc clean, four gates clean, config docs a fixed point.
@github-actions github-actions Bot mentioned this pull request Nov 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug][pi] Failed watcher registration loses cause and registration state; heartbeat recovery is not reported

1 participant