Skip to content

[Bug][pi] Failed watcher registration loses cause and registration state; heartbeat recovery is not reported #5641

Closed

Description

@xiaozhu1337

Description

In pi plugin-native mode, a transient failure of POST /__bili/watcher prints:

bili: watcher registration failed — the shared proxy may exit when its first owner does (fetch failed)

The initial request has a timeout, and attachment continues. A subsequent heartbeat can register the owner successfully, but the failure result is not retained in the returned handle and the heartbeat sender does not report successful recovery. The user is left with a lifecycle warning whose current status cannot be determined from that warning. The exception log also omits the underlying error cause, making the original failure hard to diagnose.

Observed recovery, not a confirmed proxy shutdown or data-loss incident: a later read-only health check showed the affected host PID in watchdog.watchers, with armed:true, and both health/status returning HTTP 200. No manual registration or proxy restart was performed during diagnosis.

中文摘要:pi 中 watcher 首次注册临时失败后仍继续附着,后续心跳可以补登记,但缺少已恢复提示和可查询的注册状态;错误仅显示 fetch failed,未记录底层原因。本次检查时已恢复,未证实代理误退出或数据损坏。

Environment

  • bili: 0.1.212, running commit reported by health: daa082ea; running version equals disk version.
  • Current upstream master inspected: 23a826d3788ae6bf1294dcc7ad0ef252d92ae396, also declares 0.1.212. The relevant behavior is still present in source. No plugin upgrade/restart was performed during diagnosis.
  • Usage mode: Plugin native, pi lane, several pi sessions sharing a local proxy. The proxy runs as a separate Node process.
  • Client: @earendil-works/pi-coding-agent 1.5.0.
  • Runtime: Node.js v24.22.0.
  • OS: Windows 11, build 26300, native Windows, not WSL.
  • Upstream/model: pi provider ID magpie, model ID codex/gpt-6.1-sol, Responses protocol. The backend endpoint is private; actual backend identity/context limit has not been independently verified. The failing registration request is to the local proxy, not the model endpoint.
  • Compression configuration was not changed during investigation. This report concerns registration diagnostics/state, not a compression threshold or model context-window claim.

Observed sequence and sanitized evidence

One occurrence was observed during a long-running multi-session pi workload on 2026-11-04. The network trigger is intermittent and has not been reproduced deterministically.

  1. Several pi sessions were using the same local pi-lane proxy.
  2. The watcher registration warning above appeared in the affected pi session.
  3. Inspection of the surrounding log showed a failed health probe followed by attachment to the existing proxy. Other connections also experienced resets/timeouts around that time.
  4. Later, direct read-only GET /__bili/health and GET /__bili/status returned 200, and the affected host PID was already in the watcher list.

Relevant log excerpts, with session/PID/port identifiers replaced:

11:22:50.477 [info] [v=0.1.212] [<session>] [plugin] tool compress executed via plugin (routed by body, #1685) (1066 chars, outcome=applied)
11:22:54.561 [info] [v=0.1.212] native-attach: keep http://127.0.0.1:<port> (pid <proxy-pid>): health probe failed but TCP connect succeeded — instance is BUSY (event loop saturated), not dead (#3654/#4811)
11:22:54.562 [info] [v=0.1.212] bili: attaching to running proxy at http://127.0.0.1:<port> (pid <proxy-pid>)
11:22:54.562 [debug] [v=0.1.212] [conn#<id>] tcp closed reason=error(ECONNRESET) age=83664ms reqs=1
11:22:54.569 [warn] [sess=<session>] [v=0.1.212] [<session>] [acp-proxy] upstream proxy-reset (pre-response network failure): fetch failed; retrying in 1500ms (attempt 1/3)

Later health/status observations:

health: HTTP 200
version = diskVersion = 0.1.212
watchdog.armed = true
watchdog.watchers includes <affected-host-pid>
status: HTTP 200
storage.effective = full
storage.degraded = false
storage.integrity.ok = true
secondProcess = null
splitBrain = null

The integrity value is the reported/cached status, not a new full database scan. The BUSY line is the client's health-failed/TCP-succeeded heuristic, not proof from a CPU/event-loop profile. The connection age is not a measured stall duration. The reset logs cannot be attributed specifically to the watcher POST from the available data. Its underlying cause remains unknown because the exception cause was not logged. A previous upstream stream failure and this warning should not be conflated with a permanently failed compression operation; the outcome=applied line above confirms a successful fold.

Source-level explanation

All links below are pinned to the inspected upstream commit:

  1. registerWatcherDefault, src/launcher.ts:4034–4053 sends a single POST with a 2-second timeout. HTTP 409 maps to refused; other failures map to failed. The catch prints only err.message, losing err.name, err.cause.code, target origin and owner PID.
  2. attachTo, src/launcher.ts:5062–5078 starts the heartbeat and returns { attached: true } even after failed. Only refused is reflected on the handle (refusedWatcher). There is no corresponding registration-pending/failed state.
  3. startOwnerHeartbeat, src/launcher.ts:4853–4873 only attests process.pid. For this observed pi session the heartbeat can recover registration; for foreign-owner-PID callers, it is a no-op, so this recovery mechanism must not be assumed universal.
  4. Worker sender, src/launcher.ts:4916–4938 catches transient failures but does not inspect res.ok or notify the parent of successful registration. The fallback sender likewise does not check the HTTP outcome.
  5. Server watcher route, src/server/admin.ts:554–575 adds the PID for accepted heartbeat requests as well as initial registration, explaining how later heartbeats can repair the initial failure.

Expected behavior

  • Attachment may continue using the existing proxy, but attached/transport-ready must not imply owner registration was acknowledged.
  • A transient registration failure should carry actionable, sanitized diagnostics.
  • If a later heartbeat acknowledges registration, the pending warning/state should transition to recovered, with a single host-visible recovery notice.
  • If registration remains unconfirmed, preserve that fact. A resolved fetch() returning HTTP 409/500 must not count as successful recovery.
  • Keep HTTP 409 (unarmed/manual daemon) distinct from transient transport failure. Preserve existing external-attach policy and the worker-based heartbeat design.

Proposed minimal fix / 修复说明

This is a proposal, not an implemented or tested patch:

  1. Improve the shared registration diagnostic, using existing logging/error-formatting helpers where available: sanitized origin, owner PID, error name/message and nested cause code. Report HTTP status on HTTP failures. Do not include credentials, request bodies or the private model endpoint.
  2. Retain registration outcome separately from attachment in the existing launcher/native flow. Do not turn a transient failure into an unconditional spawn/restart or direct/uncompressed fallback; this proxy is shared and may be healthy again a moment later.
  3. Optionally retry only transient registration failures a small, bounded number of times within an explicit overall budget. Do not repeatedly retry HTTP 409. This also matters for foreign-PID callers where startOwnerHeartbeat is intentionally disabled.
  4. Acknowledge recovery through the existing heartbeat worker/parent channel: inspect the HTTP result, signal the first successful acknowledgement after a pending failure, clear that pending state, and emit one recovery notice through the host-safe diagnostic channel. Apply equivalent outcome checking to the in-thread fallback. No per-heartbeat log spam and no change to watchdog death/expiry policy.

Suggested focused regression checks:

Scenario Required result
First registration transport failure, later heartbeat 200 Failure is diagnosable; pending state clears only on acknowledgement; one recovery notice; no extra proxy spawned
Persistent network failure Retry/time budget remains bounded; registration remains unconfirmed; no false success
HTTP 409 Existing refused semantics preserved; no transient retry loop or false recovered notice
Heartbeat HTTP 500 or 409 Resolved fetch does not count as recovery
Foreign owner PID No heartbeat is incorrectly started; initial failure is retained/handled explicitly
Worker failure → in-thread fallback Same acknowledgement semantics and existing cleanup/unref behavior preserved

Existing tests/watcher-registration.test.ts covers initial 200/409/500 and connection failure outcomes; it does not cover attach-after-failure followed by acknowledged heartbeat recovery. The existing dependency seams should make the state/retry tests deterministic without manufacturing a production event-loop stall.

Related issues / scope

No proxy restart, database cleanup, dependency installation or fault injection was performed. No shutdown, data loss or database corruption was established. Local paths, private endpoints and session/PID identifiers have been removed from this report.

Prepared with an AI coding assistant on behalf of the reporter; measurements and source inspection are distinguished from the untested fix proposal above.

Activity

  1. ranxianglei commented on Nov 4, 2026

    @ranxianglei
    Owner

    🤖 Powered by ework · qwen3.8-27b

    (ai: vllm-qwen/qwen3.8-27b)

    查重: 不重复。 检索 open+closed issue/PR(关键词 watcher registration / heartbeat recovery / pending / 中文变体)无同机制条目。你已列出的 #4129(注册被拒绝)、#4994(心跳确认)、#5152(心跳线程存活)与本条不同:本条的缺陷是「注册失败的结果与原因丢失、恢复不可见」,不是注册本身失败或心跳停发。

    Triage 结论(bug 成立,已在源码层核实 @ master 23a826d37):

    • 是什么问题:attach 到共享 proxy 时 POST /__bili/watcher 瞬时失败后,返回的 handle 只保留 refused(409),failed(传输故障)这一结果连同底层错误原因一起丢掉;心跳后续把注册修好时,没有任何可见信号。用户看到的永久警告与实际状态脱节。
    • 层次判断:根因不在 watchdog/心跳机制(它们工作正常——你的观测里 pid 确实被重新加回 watchers),而在三个接线缝隙:① registerWatcherDefault(src/launcher.ts:4024)只打 err.message,嵌套 cause(如 ETIMEDOUT)和 origin/pid 上下文全丢;② attachTo(src/launcher.ts:5062)只把 refused 写进 handle,failed 无状态留存;③ 心跳 worker/fallback 不检查 HTTP 结果,成功 ack 无人上报。表面现象是「警告吓人且永不消失」,病因是结果在管道里被丢弃。
    • 方案评估:你提的 4 项修复方向全部打到点上(诊断增强 / 状态分离 / 有界重试 / 经既有心跳通道上报恢复),我按此实现。其中第 3 项(有界重试)对 foreign-pid 调用方(claude lane 等无心跳路径)是唯一补救机会,会一并做;第 4 项严格区分 2xx 才算恢复,409/500 不算。不会引入无条件 spawn/restart/fallback。

    标签: bug(缺陷) pi(pi native 车道触发) scope/wiring(协议接线/启动器 attach 路径) p2(可用但有明显问题,自恢复、无数据丢失)。

    接下来动手:定向测试本地过 → PR(Closes #5641)→ CI 全局把关。

  2. ranxianglei commented on Nov 4, 2026

    @ranxianglei
    Owner

    🤖 Powered by ework · qwen3.8-27b

    (ai: vllm-qwen/qwen3.8-27b)

    查重: 不重复(open+closed 检索无同机制条目;#4129/#4994/#5152 相关但机制不同——分别是 watchdog 武装、stale proxy 轮转、心跳 worker 活性,本条是「注册失败后的可观测性与状态保留」)。
    标签: bug / pi / scope/wiring / p2(可观测性降级,心跳约 5s 内自愈,无数据丢失、无可用性损失)。

    修复完成,PR 已开: #5644 (Closes #5641)

    结构化八问

    1. 是什么问题:pi 等插件车道 attach 到已在运行的共享 proxy 时,向 watchdog 的注册请求若瞬时失败——日志只打 fetch failed 看不到真实原因、返回的 handle 丢掉这次失败结果(只记录 409)、后续心跳实际悄悄修好了却没人报告恢复,警告永远悬着。
    2. 什么原因导致的:链路上三处各自丢信息——①日志行不遍历错误 cause 链(undici 把 ECONNREFUSED/DNS 等真实原因包在 err.cause 里);②attach 路径只为 409 写 refusedWatcher,"failed" 结果没有落进 handle;③心跳发送端从不看 HTTP 状态码,服务端接受 beat 后重新登记了 PID,也没有任何通道回报父进程。
    3. 影响范围与严重程度:所有走 attach 路径的插件车道(pi/opencode/dsh 等);仅可观测性降级——proxy 靠心跳自愈,无数据丢失、无可用性损失,p2。
    4. 怎么修复的:四个机制,全部在 src/launcher.ts——① 脱敏诊断:origin + owner pid + cause 链第一个结构化 code(undici 无 code 时回退 cause message),并附「接下来会发生什么」的说明(自有 pid:心跳会重注册并报告恢复;外部 pid:保持未确认除非重试成功);② ProxyHandle 新增 watcherRegistration: ok|refused|failed,每次 attach 结果都落字段(旧 refusedWatcher 原样保留,既有消费方不受影响);③ 只对瞬时失败做有界重试:最多 3 次、间隔 500ms、最坏给 attach 增加 ~7s;409 永不重试——这是不启动心跳的外部 PID 调用方唯一的恢复机会;④ 恢复确认复用既有 worker/parent 通道:worker 检查 res.ok,首个 2xx beat 后向父进程发一次性 "ack",父进程 pending→recovered 恰好一次、经 host-safe 诊断通道打印一条恢复通知;409/5xx 应答不算恢复;in-thread fallback 路径获得同等语义。watchdog 策略不变,未新增任何 spawn/restart/fallback。
    5. 是否有回归问题:未发现。逐项核对:线上形状 {pid, hb:true} 不变;409 语义不变;refusedWatcher 兼容保留;[Bug] Watchdog takes its stale-heartbeat reading on its own event loop, so a synchronous stall kills a live proxy (parent-gone → uncompressed 400) #5152 的 worker unref 活性契约保留——且过程中发现并处理了一个 Node 陷阱:给 unref 过的 Worker 挂 message 监听器会使其端口被重新 ref、把宿主进程钉住不退出,attach 后与每次投递后重新 unref,用进程退出探针实测验证。定向套件全绿:新套件 14/14(含驱动真实 5s 间隔 worker 的端到端用例)、issue5152 6/6、issue3544 20/20、launcher 345 pass/1 skip、watcher-registration 1/1、typecheck clean;全量判定以 CI 为准。
    6. 是否破坏缓存:不涉及(无缓存/重试游标/去重键变更)。
    7. 是否改了设计方向:不涉及——按 issue 要求完全留在既有 worker-based heartbeat 设计内。
    8. 是否引入新的配置:不涉及(无新 env/config/CLI 字段)。

    行为变更披露(均为增量):两条失败日志行内容变丰富(核心句 "the shared proxy may exit when its first owner does" 逐字保留,旧 grep 仍能命中);/__bili/watcher 不可用时 attach 最坏比之前多花 ~7s(之前单次尝试即附接);每次恢复多一条恢复通知(每失败至多一次);handle 多一个增量字段。

    判定:可合入,待 CI 绿 + review。

  3. ranxianglei commented on Nov 4, 2026

    @ranxianglei
    Owner

    🤖 Powered by ework · qwen3.8-27b

    (ai: vllm-qwen/qwen3.8-27b)

    来源:#5660 源码验证结论——F2「压缩后条条带 id」前提在四车道不全部成立(opencode v1/v2 不保证,pi/dsh 有条件保证,结论表见该 issue)。按 #5660 预定分支,§3.2 容错条款激活。拟议条款文本(供 owner 并入契约正文):

    容错条款(§3.2 增补):身份信道诚实缺失——请求携带 cohort 载体(x-bili-plugin-conversation)但不带 x-bili-pf-tail-ids——按 cohort 排除处理:软降级到基线车道服务,不计入熔断连击,下一个携带完整 tail-ids 的健康轮自动回到快道。「在场但畸形 / 宽度不符 / 一致性破裂」维持 400 硬故障不变(漂移/伪造信号)。依据:§5 关键推论——增量正确性由计数几何保证、不由 id 证明;id 的角色是档案/诊断/漂移探测。

    配套测试矩阵补一行:载体会场 / tail-ids 缺失 ⇒ 静默基线、无连击(现有第 7 行「R 全 ? W>0 ⇒ 尾切片追加」互补覆盖)。

    实现定位:src/server/fastpath-v6.ts:564-569(现行为 recordFault + throw,应移入 :553 的 cohort-exclusion 同类)。发送端 ef35c98 的整表清除策略无需改动。

  4. ranxianglei commented on Nov 4, 2026

    @ranxianglei
    Owner

    🤖 Powered by ework · qwen3.8-27b

    (ai: vllm-qwen/qwen3.8-27b)

    误投说明:本线程最后一条「§3.2 容错条款」评论是 agent 编号混用导致的误投——它是 fastpath v6 契约 #5643 的条款提案,与本线程的 pi watcher 注册问题无关。已在正确线程(#5643)重新发布,请忽略/删除该评论。

  5. added a commit that references this issue on Nov 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions