Repository navigation
[PR] fix: surface failed watcher registration cause/state and report heartbeat recovery - #5644
Conversation
…4599) Master 9dc4c94 red on test (windows-latest, 22, 2) only: the two SSE abort cells exhausted the 10s teardown poll (closed=0, ~11s each) while the same code went green on ubuntu22/24 and windows24 shard-2 siblings (pass durations 23-352ms). The audit mandated by the constant's own note found no teardown hang: relay.ts arms res.on('close') -> clientAbort.abort() synchronously with no linger/backstop on this path, and no commit since d244fff touched destroy/abort wiring. Residual latency is runner-side event-loop starvation on loaded Windows runners (same family as the recorded 3s/6s eras). Platform-scoped response: win32 doubles to 20s; every other platform keeps the tight 10s gate so a genuine hang still fails fast.
…beat recovery (#5641) A transient POST /__bili/watcher failure on attach printed a bare "fetch failed" line, dropped the outcome from the returned handle (only 409 was retained), and then self-repaired silently via the owner heartbeat with no trace. - registerWatcherDefault diagnostics carry sanitized origin + owner pid, the error name/message plus the first structured cause code (or cause message for undici errors that carry no code), and an outcome-specific tail (own pid: a later heartbeat re-registers and reports recovery; foreign pid: stays unconfirmed unless a retry succeeds). - ProxyHandle gains watcherRegistration ("ok"|"refused"|"failed") for every attach outcome; refusedWatcher is kept for existing consumers. - attachTo retries transient failures only (never 409): up to 3 attempts, 500ms backoff, worst-case ~7s extra attach latency. - The heartbeat worker inspects res.ok and posts a one-shot "ack" after the first 2xx beat; the parent transitions pending->recovered exactly once through the host-safe diagnostic channel. The in-thread fallback path acks via the post() boolean result. A resolved fetch returning 409/5xx never counts as recovery. Attaching the "message" listener re-refs the worker port (Node MessagePort semantics), so unref is re-asserted at attachment and after every delivery to keep the #5152 liveness contract. No watchdog death/expiry policy change; no spawn/restart/fallback added; no config surface change. Tests: tests/issue5641-watcher-registration-recovery.test.ts covers the issue six-row regression table.
📦 Built Package ArtifactBranch: Latest build — install this exact versionnpm install -g billion-context@0.0.8100-pr.5644.6e63e8fThis exact version is immutable and reproducible. The rolling Recent builds (commit → version)
Download artifact
tar xzf billion-context-pr5644.tgz
npm install -g packageInstall instructions are refreshed on each push. |
#5428 made inline the pi lane default; this suite documents the spawn contract (instance record + proxy origin, per-request stamps, the shared-proxy delegate-child channel, /__bili/plugin/status over HTTP) and was not updated with the flip. Under the inline default its delegate-child stamp assertions get 0 stamped child requests (child traffic reaches upstream unstamped, conv=null) and the /acp instance- record assertion fails structurally. Pin BILI_PI_INLINE=0 for the whole suite so the spawn-posture coverage stays deterministic; inline posture is covered by tests/pi-inline-ws.test.ts / tests/pi-native.test.ts.
…r-registration-recovery
(ai: vllm-qwen/qwen3.8-27b) 查重: 不重复 — 无其他开放 issue/PR 处理 watcher 注册诊断/恢复;唯一相邻的 #5625 是 win22-s2 的 CI 抖动修复,与本 PR 无关。 Review of PR #5644 — 独立审核结论方向关(①):原 issue #5641 报三个缺陷——失败丢失底层原因、handle 丢弃 "failed" 结果、心跳补登记后无恢复信号。PR 三项全中,外加有界重试(理由成立:foreign-pid 调用方如 claude 车道没有心跳兜底,重试是其唯一修复路径;own-pid 则缩短未登记窗口)。修在决策点(注册结果的处理与上报),不是症状掩盖。版本核对:报告方 v0.1.212(daa082e),master head 23a826d 同样声明 0.1.212 且缺陷机制在 base 上确认存在。✓ 回归审查(②)— 触碰路径的既有行为清单,逐项判定:
无无意回归。高危模式逐项过:无 fail-fast→吞错切换(注册失败仍降级不阻塞 attach,与旧行为一致)、无阈值/时序漂移超出披露范围、无输出格式变化超出披露范围、无缓存/去重策略触碰。 代码审查(③):逻辑干净。 测试有效性(④):真测试。断言有牙且双向:精确计数(calls===3、sleeps=[500,500])、精确日志子串、wire 形状 deepEqual、负向行齐备(500/409 应答不算恢复、falsy post 不算 ack、refused/ok 都不重试)、幂等(恰好一条通知、settled 后无刷屏)、stop() 冻结 fallback 心跳、e2e 走真 ensureProxyRunning + 生产 5s 间隔 worker。临时端口 listen(0),无端口假设。这个 bug 回归时抓得住。 本地验证(定向,非全量):rebased head 上 CI 状态与处置:c8270e75c 上 47 项检查,唯一红 = 小观察(不拦合并,同一通知精度家族):多 attach 拓扑下恢复通知的归属可能偏移——① foreign-pid(claude 车道)注册失败也会 mark pending,但 foreign pid 不起心跳、stop() 永不到来,该条目驻留至进程退出(几字节×origin 数,无功能影响);若同进程另一车道随后以 own-pid 对同一 origin 成功 attach,其 beat 的 ack 会清掉这条 pending 并发出「recovered (owner pid )」通知,而服务端实际重加的是 beat 方的 pid—— 行为变更披露核对(Deliverables 第 4 条):PR 描述的六项变更逐一与实现比对,全部属实,无隐藏漂移——失败日志行扩写(origin+pid+cause+tail)、endpoint 不通时 attach 时延 结构化八问
判定:可合并 —— 代码与测试质量达标,行为变更全部如实披露;唯一待办是新 head 71c2675 的 CI 最终绿(预期:继承性 win22-s2 flake 已被 ci-fixes 上的预算修复覆盖)。上述小观察建议作为后续精修或并入你已有的车道计划,不阻塞本 PR。 |
(ai: vllm-qwen/qwen3.8-27b) CI 终态确认:新 head 审查结论维持:可合并,无遗留阻塞项。合并后若处理那条小观察(pending 条目在 foreign-pid/多 attach 拓扑下的通知精度),按「同一机制一个 issue」归入 watcher 注册家族即可。 |
(ai: vllm-qwen/qwen3.8-27b) 七问(本次 auto-fix run,attempt 1/3)
收工判定:head 71c2675 上 30 项 check 全部 completed、零失败(原红的 win22-s2 已转绿),本次红 CI 修复完成,PR 待人工合并。另在 #4599 留了一行收敛记录。 |
…ies (#5641) Review nit (comment-only): the doc comment listed three ways a pending-recovery entry dies, but a foreign-owner-pid entry has no heartbeat to stop it and lives until process exit or a later direct-ok attach to the same origin. State that so the residual is visible in-code. No behavior change.
ranxianglei
left a comment
There was a problem hiding this comment.
(ai: vllm-qwen/qwen3.8-27b)
APPROVE(comment-review 形式提交——GitHub 禁止提交者正式 approve 自己的 PR,结论不受影响)。 独立全量审阅(diff 逐 hunk 对 master 23a826d37 = 当前 base 通读,不依赖 issue 侧结论与线程内先前审核);CI 在 head 71c267515 全绿(镜像缓存:30 checks = 17 success + 13 skipped,0 failed/pending)。
结构化八问
- 是什么问题:attach 到已在运行的共享 proxy 时,watchdog 注册请求(
POST /__bili/watcher)瞬时失败后——日志只打fetch failed看不到真实原因、返回的 handle 丢掉这次失败结果(只有 409 有标志)、后续心跳实际悄悄修好了注册却没有任何可见信号,警告永久悬着与实际状态脱节。 - 什么原因导致的:链路上三处信息丢弃点——①
registerWatcherDefault只打err.message,undici 的嵌套 cause(ECONNREFUSED/DNS 等)与 origin/pid 上下文丢失;②attachTo只为 409 写refusedWatcher,"failed" 结果无处落地;③ 心跳 worker/fallback 发送端从不看 HTTP 状态码,服务端在接受的 beat 上重新登记 PID 后没有通道回报父进程。修复对准三个丢弃点本身,不是治标。 - 影响范围与严重程度:所有走 attach 路径的插件车道(pi/opencode/dsh/claude 等);仅可观测性降级——心跳 ~5s 内自愈,无数据丢失、无可用性损失(p2)。
- 怎么修复的:四个机制全部在
src/launcher.ts——① 脱敏诊断(origin + owner pid + cause 链首个结构化 code,无 code 回退 cause message;仅 name/message/code,不落 body/凭据);②ProxyHandle.watcherRegistration: ok|refused|failed每次 attach 结果都落字段(旧refusedWatcher原样保留);③ 只对瞬时失败做有界重试(3 次、500ms backoff、409 永不重试)——这是不启动心跳的外部 PID 调用方唯一的补救机会;④ 恢复确认复用既有 worker/parent 通道:worker 检查res.ok,首个 2xx beat 后发一次性"ack",父进程 pending→recovered 恰好一次、经 host-safe 诊断通道输出;409/5xx 应答永不算恢复;in-thread fallback 同等语义(post 严格 resolvetrue才计)。独立核实:三条 attach 路线(waitForOtherStarter/adoptServingTwin/rediscoverAfterChildExit)全部收敛进attachTo,无绕过;spawn 路径不需要注册(子进程原生跟踪 BILI_PARENT_PID),不受影响。 - 是否有回归问题:未发现。逐项清点既有行为:wire 形状
{pid}/{pid,hb:true}不变(测试逐字钉住);409→refused 语义与 claude-mcp-entry/claude-native-bootstrap 消费链不变;registerWatcherDefault「永不抛」契约不变;#5152 worker unref 活性契约保留并加固(挂 message 监听器会 re-ref port——监听器注册完成后及每次投递后重新 unref,顺序核实正确);#3544 injected-post seam 保留(既有测试的 post resolve undefined 不会被误判为 ack);OwnerBeatWorker 接口扩宽向后兼容(CI typecheck 全绿);spawn 路径 5 处 startOwnerHeartbeat 调用点为纯上下文行未动;kernel/ 未触、config 面未动、version 字段未动。行为变更披露(均为增量):两条失败日志行内容变丰富(核心句 "the shared proxy may exit when its first owner does" 逐字保留,旧 grep 仍命中);/__bili/watcher不可用时 attach 最坏比之前多花 ~7s(之前单次尝试即附接)——有界、仅失败路径发生、正常路径零开销;每次注册失败事件多一条恢复通知(至多一条)。 - 是否破坏缓存:不涉及(无缓存/重试游标/去重键变更)。
- 是否改了设计方向:不涉及——完全留在既有 worker-based heartbeat 设计内,未新增任何 spawn/restart/fallback,watchdog 策略不变。
- 是否引入新的配置:不涉及(无新 env/config/CLI 字段)。
测试有效性
真测试,断言有牙。14 个用例把 issue 场景表六行全覆盖且正反方向齐备(gate 双向:409/500 beat 不算恢复、falsy post 结果不算 ack、干净注册不重试、409 永不重试);用真实 HTTP stub(listen(0) 临时端口)+ 真实 Worker(非仅 fake);端到端用例驱动生产 5s 间隔的 worker 走真实 ensureProxyRunning attach 路径,spawnImpl 直接 throw 钉死「不得多开 proxy」;确定性 seam(deps.sleep/registerWatcher/injected post)保证 CI 可复现。四个机制任何一个回归都会让对应行变红。
Diff 整洁度
主提交只触 src/launcher.ts + 新测试文件;其余为 merge ci-fixes 带入的测试调整(e2e pi 套件钉 pre-#5428 spawn posture、win32 teardown 预算 10s→20s 平台限定),均带 provenance 注释、#4599 哨兵收敛、已逐一目检,test-only。无无关改动、无空白噪音、无 lockfile 漂移。
已知残留(非阻塞,记录在案)
「多车道共享同一宿主进程」的罕见拓扑下:外部 PID attach 失败留下 pending entry 后,若原 own-pid worker 中途死亡,fallback 循环的首个 2xx beat 可能发出一条标注外部 pid 的恢复通知(实际被重新 attest 的是 process.pid)——触发需三重巧合叠加,后果是一条误导日志行,功能性 attest 不受影响;另同一 origin 同进程内的第二次 attach 失败(worker 一次性 ack 已消费)不会再有通知(功能上仍由 beat 自愈)。两者属新功能的设计取舍,代码不动,留给作者后续判断。
已顺手修:watcherPendingRecovery 文档注释未说明 foreign-pid entry 的生命周期(无心跳可停它,活到进程退出或同 origin 下一次 direct-ok attach)+ commit 30b18d156(comment-only,无行为变更)。
判定:可合入。
[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b) Review 完成,结论:可合入(APPROVE,comment-review 形式提交——GitHub 禁止提交者正式 approve 自己的 PR)。
|
…5644) win22-s2 red on head 30b18d1: 'a delta re-claiming a live canonical id' read ['u1'] where ['a1','u1','u2'] was expected — the HTTP turn flow was correct (200/400/appliedSeq all asserted fine); only the durable rows were missing. Mechanism: the lane wrap-up arms a debounced save AFTER the ack (incremental-lane.ts), whose async write chain fsyncs behind the test's fixed 60ms settle sleep; under loaded windows-latest runners the fsync outran the budget and rows() saw the pre-turn snapshot. Same code green on the previous head (71c2675) and on every other platform/shard. Replace the fixed sleep with the store's own landing verdict (afterNextFlush, #5155/#4788 D2 — the same signal the production pointer upkeep already awaits): deterministic when the save is slow, and a false verdict fails loud at the verdict instead of as a confusing missing-row diff if a save genuinely fails.
(ai: vllm-qwen/qwen3.8-27b) 七问(本次 auto-fix run,attempt 1/3)
收工判定:head 50318a4 上 30 项 check 全部 completed、零失败,本次红 CI 修复完成,PR 待人工合并。另在 #4599 留了备注:三个兄弟测试文件(seq-interval-lane / incremental-lane / issue5155-incremental-per-turn)共享同一固定 sleep settle 模式,当前无红,收敛与否留给哨兵裁量。 |
ranxianglei
left a comment
There was a problem hiding this comment.
(ai: vllm-qwen/qwen3.8-27b)
APPROVE(第 2 轮增量复审,comment-review 形式——GitHub 禁止提交者自批)。 本轮 delta = 我上轮审阅状态 30b18d156 之上新增的唯一一个提交 50318a49b(test-only:tests/msgid-lane-flow.test.ts +14/-4);CI 在新 head 50318a49b 全绿(镜像缓存:17 success + 13 skipped,0 failed/pending)。上一轮对主修复(launcher.ts 四机制)的完整八问结论继续有效,本轮只审增量。
增量独立核实(不依赖提交信息自述)
- 根因合理性:win22-s2 在
30b18d156红于'a delta re-claiming a live canonical id'(rows 读到 pre-turn 快照)。该测试与本 PR 的 launcher 改动无任何耦合——机制是 #5155 的 debounce 落盘(15ms)+ fsync 在负载型 windows-latest runner 上跑赢了测试里写死的 60ms settle sleep。证据链自洽:同一代码在上一个 head 与其余全部平台/shard 均绿。这是典型的「sleep 预算 vs 完成信号」竞态,修法方向正确。 - 新 settle 语义逐行核实(src/persist.ts:3064
afterNextFlush):无 pending flush 时立即 resolve true(incremental-lane.ts:916 注释明示,生产 pointer upkeep 已依赖此语义)→ 不会挂死;save 真失败时 waiter 以失败判决 drain → 测试assert.ok(ok)在判决点 loud-fail,而不是留下令人困惑的 missing-row diff。严格比旧行为更强(旧 settle 恒真),且给测试加了牙。 - 会话键核对:
afterNextFlush("conv-msgid-1")与测试实际使用的x-bili-plugin-conversation: conv-msgid-1/ SQL 查询键一致,无错键等待。 - 无重复修复:origin/master 与 origin/ci-fixes 上该文件历史一致(最新 f13707f),均不含此改动——本分支独有,合入后随 master 收敛;按几行级小修政策,CI 红小修以标注 commit 落在当前 PR 是允许的形态。
- diff 整洁度:单文件、纯测试、注释同步更新、无空白噪音。
结构化八问(本轮增量范围)
- 是什么问题:msgid-lane-flow 测试用固定 60ms sleep 等 debounce 落盘,负载型 Windows runner 上 fsync 跑赢预算导致读到旧快照(CI 红,与本 PR 功能无关)。
- 什么原因导致的:测试确定性依赖时间预算而非完成信号(§7.2「no environmental luck」的反例)。
- 影响范围与严重程度:仅 CI 稳定性(win22-s2 偶发红),不影响产品行为。
- 怎么修复的:settle 改为 await 存储自身的落盘判决
afterNextFlush(sessionId)并断言其真值——慢则等、真失败则在判决点响亮失败。 - 是否有回归问题:无。产品代码零改动;测试语义只强不弱(恒真 sleep → 带断言的完成信号)。
- 是否破坏缓存:不涉及。
- 是否改了设计方向:不涉及。
- 是否引入新的配置:不涉及。
判定:可合入。 两轮合计:主修复四机制成立、回归清点干净、测试有牙、diff 整洁;本轮增量是一处正确的测试确定性修复,无新增风险。
[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b) 第 2 轮增量复审完成(head
|
📦 Merged Build PublishedThis merge to master was published as npm install -g billion-context@0.0.1035-master.e4c50ef # this exact build
npm install -g billion-context@master # always the newest merged build
|
…mpress gate, ranxianglei#5686 dsh drift consolidation, ranxianglei#5696 model-scoped keyring 429 Ride-along upstream work that landed after c9ba485: - ranxianglei#5683 forbid unprompted / out-of-bounds compression - ranxianglei#5686 dsh drift consolidation (last native-compaction leak window) - ranxianglei#5696 model-scoped KeyRing rate-limit holds - ranxianglei#5661 incremental-lane credential headers, ranxianglei#5656 single-shot sidecar - ranxianglei#5644 watcher registration recovery, ranxianglei#5635 restart survival Overlap with this lane is limited to regenerated artifacts (CONFIGURATION*.md) and to the shared relay/core seams, both auto-merged. No source conflict. Verified after rebuilding kernel: issue5175 58/0, tag-echo 79/0, issue2190-degen-close 32/0, golden-storage-render 5/0, storage-engine-proofs 12/0, native-compact-terminal 1/0, plugin-passthrough-tag-strip-chat 20/0, plus the newly arrived ranxianglei#5683/ranxianglei#5686/ranxianglei#5656/ranxianglei#5658/ranxianglei#5631/ranxianglei#5641 suites all green. tsc clean, four gates clean, config docs a fixed point.
Closes #5641
What was broken
When pi (or any plugin lane) attaches to an already-running shared proxy, it registers its own PID with the proxy's watchdog via
POST /__bili/watcherso the proxy cannot exit while an owner is still alive. If that registration request fails transiently:bili: watcher registration failed — … (fetch failed)— the underlying cause (connection refused, DNS failure, timeout, …) was lost;What changed (all in
src/launcher.ts)registerWatcherDefault+ new exportedformatWatcherError): the failure lines now carry the origin, the owner PID, the HTTP status when the request completed, and the first structured cause code from the error chain (falling back to the cause message for undici errors that expose no code, e.g. DNS lookups). A short tail states what happens next: own PID → "a later heartbeat will re-register us and report recovery"; foreign PID → "no heartbeat runs … stays unconfirmed unless a retry succeeds". No credentials, request bodies, or private endpoints anywhere in it.ProxyHandlegainswatcherRegistration: "ok" | "refused" | "failed", set on EVERY attach outcome;refusedWatcheris untouched for existing consumers. Attached/transport-ready no longer implies watchdog-acknowledged.res.okand posts a one-shot"ack"to the parent after the FIRST 2xx beat; the parent transitions pending→recovered exactly once and prints ONE host-visible notice through the existing diagnostic channel. A resolved fetch returning 409/5xx never counts as recovery. The in-thread fallback (worker spawn failure / mid-flight crash) gets equivalent semantics via thepost()boolean result. No per-heartbeat logging, no watchdog death/expiry policy change, no spawn/restart/fallback added.One Node runtime gotcha handled along the way: attaching a
messagelistener to the (unref'd) heartbeat worker RE-REFS its MessagePort and would pin the host process open —unref()is re-asserted at attachment and after every delivery, preserving the #5152 liveness contract (verified with dedicated process-exit probes).Behavior changes (disclosed)
bili: watcher registration failed — the shared proxy may exit when its first owner does (fetch failed)… returned HTTP <status> — …(no origin/pid)/__bili/watcheris downProxyHandlerefusedWatcherset only for 409watcherRegistrationset for all outcomes (additive field)"ack"parent message after first 2xx beat (worker-internal only; wire shape{pid, hb:true}unchanged)bili: watcher registration recovered at <origin> (owner pid <pid>) — …No change to watchdog arming/expiry, 409 semantics, or the external-attach policy.
Config surface
N/A — no new env vars, config keys, or CLI flags.
Tests
New
tests/issue5641-watcher-registration-recovery.test.ts(14 tests) covers the issue's six-row regression table: transport failure → later 2xx beat → single recovery notice + zero spawns; persistent failure → bounded retries, stays unconfirmed; 409 → refused semantics preserved, never retried; 500/409 heartbeat responses do NOT count as recovery; foreign PID → no heartbeat started, outcome retained; worker crash → in-thread fallback with the same ack semantics. Plus an end-to-end attach test driving the real 5 s-interval worker.Targeted local runs green: this suite (14/14, clean process exit),
watcher-registration,issue5152-owner-beat-survives-host-stall(6/6),issue3544-stale-proxy-rotation(20/20),launcher(345 pass / 1 pre-existing skip); typecheck clean. Full-suite verdict: CI on this head.