Repository navigation
Description
环境 / Environment
- billion-context: 0.1.212(latest;日志
[update] current=0.1.212 latest=0.1.212 tag=latest (up to date)) - DSH: @deepseek-ai/dsh-desktop 0.2.2-rc.1(Windows,Electron 安装版)
- Node: v24.15.0,OS: Windows 10.0.26300
- 日志:
C:\Users\lenovo\.local\state\billion-context\bili.log
症状 / Symptom
-
某个大会话的每一轮都返回 422:
{"type":"server_error","code":"bili_gate_oversize_refusal_fallback","message":"billion-context refused this request locally: the bili proxy did not come up ... core text (~1,011,239 tok floor estimate from 4,044,954 B / 4) exceeds the model context window (1,000,000)"} -
同一时间窗内,新建会话、重连模型也一并失败。
-
约 7 分钟后自愈,但代理重新落在 18788(启动参数
--port 18788),18787 永远不再被绑定;lanes已被 +1 重写([launcher/lane] A serving lane is abandoned on a build change or when busy: the launch ladders +1 and rewrites lanes.<lane>, leaving two instances on one sessions dir #4568 族)。 -
恢复后旧会话仍极慢(每轮本地 0.9-7 s),缓存命中从 96-99% 掉到 59%。
时间线 / Timeline(2026-11-07,bili.log 原文)
15:10:05.737 [info] native-attach: keep http://127.0.0.1:18787 (pid 148832): health probe failed but TCP connect succeeded -- instance is BUSY (event loop saturated), not dead (#3654/#4811)
15:10:05.738 [info] native-attach: skip http://127.0.0.1:18787 (pid 148832): incompatible (code-fingerprint-mismatch)
15:10:32.670 [warn] [dsh-client] proxy bootstrap failed -- model traffic goes direct (uncompressed): bili: proxy child exited before becoming healthy (code 1) -- port 18787 accepts TCP connections but never answers /__bili/health within ...
15:10:47.705 [error] [native-client] bili-native: proxy not ready -- model requests go direct (uncompressed) -- oversized request refused locally instead
15:10:47.705 [error] [native-client] bili-native: oversized request refused locally instead of direct send -- core text ~1,011,239 tok floor (4,044,954 B / 4; media 20,949,700 B excluded, #4111) exceeds the 1,000,000 tok window
约 15:18-15:21:僵尸进程 148832 自行退出;新实例以 18788 启动并健康:
GET http://127.0.0.1:18788/__bili/health
{"ok":true,"upstream":"https://api.anthropic.com","version":"0.1.212","commit":"daa082ea","pid":138000,...}
GET http://127.0.0.1:18787/__bili/health -> connection refused(至今仍无 listener)
恢复后同一个大会话的代价(最后一行说明压缩其实已经生效):
15:23:23.756 [warn] [fold-reconcile] resent history drifted: 690 covered id(s) missing with no anchor match -- mutation (content edit invalidates content-hash refs, fold silently lost) or client-side deletion/truncation (benign, message no longer on the wire) (#1921) (~482 tok unfolded in churn region)
15:23:29.755 [info] request: 2517 msgs, inbound=23.8MiB, outbound=4.7MiB, local=7035ms
15:23:29.755 [warn] slow-session: 2517 msgs / inbound=23.8MiB took local=7035ms -- per-turn O(messages) synchronous scans are starving the event loop (#4528)
15:25:57.240 [info] [acp-usage] input=71123 cached=41728 (cache hit 59%) fold=new
15:26:01.514 [info] request: 2521 msgs, inbound=23.9MiB, outbound=262.1KiB, local=861ms
根因链 / Root-cause chain
- 僵尸实例占端口:一个旧代码指纹的代理进程(pid 148832)仍持有 18787;TCP 能 connect,但
/__bili/health永不响应(事件循环饱和)。 - 健康判据判成"忙"而不是"死":
keep ... BUSY (event loop saturated), not dead (#3654/#4811)—— 不 kill、不接管。 - fingerprint 分支紧接着又把它 skip 掉:
skip ... incompatible (code-fingerprint-mismatch)—— 既不用它,也不终止它。两条规则叠加 = 端口被永久占死且无人清理。 - 新实例 bind 失败:
proxy child exited before becoming healthy (code 1) -- port 18787 accepts TCP connections but never answers /__bili/health。连续多轮 spawn 全失败。 - 降级路径对超大会话不可用:无代理则直连不发压缩,但核心文本约 1.01M tok > 1M 窗口,于是
refused locally(DSH's unattributed over-limit gate prices base64 images as text (bytes/4) — media-heavy requests are falsely rejected wi | dsh 无归属超限门把 base64 图片当文本计价(bytes/4)——媒体重的请求被假拒 422,上游本会正常接受 #4111)。这是插件唯一正确的选择,但它把"降级变慢"变成"完全不可用",且错误码bili_gate_oversize_refusal_fallback对用户零可操作性。 - 恢复后 Fold-state reconciliation: classify resent-history divergence (edit/fork/new/system-only) and make compression blocks re-anchorable chain units #1921 折叠漂移未重锚:
690 covered id(s) missing with no anchor match ... fold silently lost→ 缓存命中 59%,首轮重发 3.8-4.7 MiB。
为什么 #4825(self-heal one-shot bootstrap failures)没覆盖到这里
- [PR] fix: dsh lane — keep saturated proxies attachable instead of spawning into EADDRINUSE; self-heal one-shot bootstrap failures (#4811) #4825 解决的是 EADDRINUSE 时不要 spawn 以及单次 bootstrap 失败自愈;
- 本例失败在 health 探测层:TCP 握手成功但 HTTP 不应答,被保守分支判为
BUSY([Bug?] Under high load, health check timeouts appear to be misinterpreted as "proxy unavailable": large session requests | [Bug?] 高负载下 health 探活超时疑似被当成「代理不可用」:大会话请求被本地拒发(0.1.199 / plugin native) #3654/dsh lane: BUSY-drop → EADDRINUSE detour → adopt-twin race is silently lost ~half the time, and a one-shot bootstrap failure then becomes permanent #4811),因此不进入覆盖/接管逻辑; - 而 fingerprint-mismatch 分支只 skip、不 terminate,所以"忙且不兼容"的实例既是端口占用者,又是被完全忽略的进程。
- 这是两条独立策略的交汇点:任意一条单独看都合理,叠加后形成死锁。
建议修复 / Suggested fixes
- fingerprint-mismatch 分支应
terminate()而不是skip():同 lane、同端口、不同代码指纹的实例,唯一合理动作是让它退出(SIGTERM,2 s 后 SIGKILL),否则端口永久泄漏。 BUSY判定要有次数/时限上限:连续 N 次(或 T 秒)health 超时即降级为dead并强制接管。当前"饱和不算死"在长会话场景下等于"永不恢复"。- spawn 失败区分 EADDRINUSE 与其它 exit code,对 EADDRINUSE 做"清理占用者 → 重试一次"的原子序列,而不是把 body 交给直连路径。
bili_gate_oversize_refusal_fallback应携带可操作信息:至少给出"核心文本 4.0 MB / 媒体 20.9 MB / 代理未就绪"三元组与建议动作。- lane 端口粘滞(18787 → 18788 后不回退):lanes 重写为 +1 时应在日志与健康端点显式标注 canonical port 失效,避免外部健康探测误判为"代理未启动"。
- Fold-state reconciliation: classify resent-history divergence (edit/fork/new/system-only) and make compression blocks re-anchorable chain units #1921 fold 漂移要重锚而非静默丢弃:
690 covered id(s) missing时应以最近存活锚点重建覆盖区间,否则每次长会话 resume 都要重发数 MB,缓存命中断崖下跌。
关联 / Related
- Event-loop-watchdog self-exit + split-brain under 30–45 MB sessions: proxy disappears mid-request (422 oversize refusal) and two instances commit the same session #5507(30-45 MB 会话下 watchdog 自退 + split-brain,proxy 中途消失导致 422 oversize)—— 高度同族,但本例是 fingerprint × BUSY 双分支死锁,不是 watchdog 自退。
- [PR] fix(agent): hold in-flight launcher respawns under the spawn budget in dsh/omp/opencode lanes #5510(PR:hold in-flight launcher respawns under the spawn budget)
- #4516's oversize circuit breaker has a dual false positive: B/4 token estimation overestimates CJK content by 7x + windo | #4516 的 oversize 熔断存在双重误判:B/4 token 估算对 CJK 内容高估 7 倍 + 窗口比较未使用实际请求模型的上下文窗口 #5871([pi] Direct connection fallback after proxy disconnect: no oversized rejection, no self-healing, no visible alerting (al | [pi] 代理失联后的直连兜底无 oversized 拒绝、无自愈、无可见告警(opencode/dsh 都有)—— 1.06M token 原始历史被直接送进上游 #4516 的 oversize 熔断双误判:B/4 对 CJK 高估 7 倍)
- omp/opencode lanes: oversized history egresses uncompressed when the proxy is down — direct-send window guard still unarmed (#5025 family residual) #5032(proxy 宕机时超大历史无压缩直发,窗口守卫未武装)
- 0.1.212 still reproduces the structural issue from #4528: single-request local preprocessing peaks at 144.7s; conversati | 0.1.212 仍复现 #4528 的结构性问题:单请求本地预处理最高 144.7s,关闭插件后对话明显变快 #5869(0.1.212 仍复现 Long sessions starve the event loop: a 4,600-message session blocks the proxy and the web UI for 7–50 s per turn #4528,单请求本地预处理 144.7 s)
- 0.1.212: per-turn ingest is still O(history) — 96.5% of proxy CPU in StatementSync.run, event loop starves with 8 concurrent sessions (#4528 / #5469 follow-up) #5744(per-turn ingest 仍 O(history),96.5% CPU 在 StatementSync.run)
- [PR] fix: dsh lane — keep saturated proxies attachable instead of spawning into EADDRINUSE; self-heal one-shot bootstrap failures (#4811) #4825(已关闭,EADDRINUSE 自愈)、[launcher/lane] A serving lane is abandoned on a build change or when busy: the launch ladders +1 and rewrites lanes.<lane>, leaving two instances on one sessions dir #4568(lane +1 后双实例)、Fold-state reconciliation: classify resent-history divergence (edit/fork/new/system-only) and make compression blocks re-anchorable chain units #1921、DSH's unattributed over-limit gate prices base64 images as text (bytes/4) — media-heavy requests are falsely rejected wi | dsh 无归属超限门把 base64 图片当文本计价(bytes/4)——媒体重的请求被假拒 422,上游本会正常接受 #4111、[pi] Direct connection fallback after proxy disconnect: no oversized rejection, no self-healing, no visible alerting (al | [pi] 代理失联后的直连兜底无 oversized 拒绝、无自愈、无可见告警(opencode/dsh 都有)—— 1.06M token 原始历史被直接送进上游 #4516
复现要点 / Repro hints
- 让一个会话长到 2500+ msgs、20 MB+ 媒体(本例 2517 msgs、inbound 23.9 MiB、582+ 张截图);
- 升级插件版本,但保留上一版本的代理进程,让它持有 18787 且事件循环饱和;
- 触发一轮请求,预期日志序列:
keep ... BUSY→skip ... code-fingerprint-mismatch→proxy child exited (code 1)→oversized request refused locally。
旁证:同机同版本下,小会话(80 msgs、outbound 210-230 KiB)全程 96-99% 缓存命中、零报错。 因此这不是环境或网络问题,而是"大会话 × 进程生命周期"的组合缺陷。
Activity
[-][Bug][dsh] 0.1.212: 僵尸代理占端口 + fingerprint 分支只 skip 不 terminate -> 超大会话持续 422(BUSY 判定 × EADDRINUSE 死锁)[/-][+][Bug][dsh] 0.1.212: Zombie proxy holds port + fingerprint branch only skips without terminating → oversized sessions stu | [Bug][dsh] 0.1.212: 僵尸代理占端口 + fingerprint 分支只 skip 不 terminate -> 超大会话持续 422(BUSY 判定 × EADDRINUSE 死锁)[/+]ranxianglei commentedon Nov 8, 2026
(ai: qwen/qwen3.8-27b)
查重:不重复。 已检索 open+closed issue/PR。最接近的家族与本单的区别:
结论:决策表缺「同车道 + 旧/异字节 + 无健康应答(BUSY)+ 占规范端口」这一格,各家族都没覆盖到它。
版本核查:你日志里的
commit daa082ea= tag v0.1.212。我读了该 tag 的代码(非 master 推测):三个裁决面与当前 master 逐行一致(git diff v0.1.212..master -- src/listen-policy.ts src/port-owner.ts src/event-loop-watchdog.ts= 0 行),机制在你的构建里确实存在,不是版本漂移误读。层次判断:你的根因链成立,且比表面更深一层——不是某个分支写错,而是三条路径各自合理、合起来无人负责:
pickAttachable对 fingerprint 不符者 skip(Launcher proxy reuse (ensureProxyRunning) breaks per-run isolation: shared-fate lifecycle + no code/client identity in compatibility check #1225 不信任未证明字节);[Bug] Plugin bootstrap spawns a new instance while the old one still holds the port and drops proxying (goes direct): la | [Bug] 插件 bootstrap 在旧实例仍占端口时新起实例并放弃代理(goes direct):大会话随即以未压缩历史撞上游上限(400 Range of input length / 413) #4125 的上行收养豁免要求健康应答里的 version——BUSY 拿不到 version,豁免不生效(src/launcher.ts:4313);rotateStalePredecessor明确排除无 health 的候选——注释原文「BUSY candidates carry no health … never rotation targets」(dsh lane: BUSY-drop → EADDRINUSE detour → adopt-twin race is silently lost ~half the time, and a one-shot bootstrap failure then becomes permanent #4811 有意设计,src/launcher.ts:4916-4918);findNewerSameLaneHolder同样要求健康应答。→ 占着 18787 的旧实例没有任何一个 actor 会去解决它,直到它自己死。而 event-loop watchdog 只在连续阻塞 ≥5 分钟才自杀(event-loop-watchdog.ts:45);你这种「切片间让出 event loop 的饱和态」永远触发不了 → 僵尸可无限期存活。这就是「小会话正常、超大会话持续 422」的组合条件:只有大请求才能把 loop 压到连 1.5s 健康探针都答不出来的程度。
附带确认:恢复后车道钉在 18788 不回 18787,就是 #4568 族的 sticky +1 棘轮——失败 spawn 期间 child 爬到 18788 并 settle 了粘性端口。
方案评估:你的建议 1-3 指向同一机制(生命周期死锁裁决),我会合并为一个修复而不是三处补丁:对「同车道 + fingerprint 不符 + BUSY + 占目标端口」的 holder 引入有界耐心 + 强制接管——跨 bring-up 周期累计观察(首次观察时间戳 + 观察次数),超过窗口(暂定 10 分钟 = 2×watchdog 硬超时,硬编码常量不加配置项)后先试优雅 shutdown(有界 3s),仍占着就按 killAbandonedChild 同款方式强杀(POSIX SIGKILL / win32 taskkill /F /T)、确认端口释放后继续正常 spawn。选「耐心+接管」而非你的建议 1(立即 terminate)的原因:BUSY 可能是慢但活着的实例正在做合法的多分钟压缩,立即杀会丢在途工作;窗口把最坏不可用从 ∞ 限到 10 分钟,数据损失上限 = 受害实例在途的一轮(session 每轮落盘)。
你的建议 4(拒绝消息可操作性)、5(车道粘滞可观测性)、6(#1921 fold drift 重锚)属于不同机制/不同症状,主线 PR 落地后我会分别带来源标记立项,不在本 PR 扩范围。
交付计划:分支 + PR(含决策表双向测试 + ledger 测试 + 全路径注入缝测试),本地只跑定向套件,全量回归交 CI。开始实现。
TOSKA1226 commentedon Nov 8, 2026
感谢定位,三条裁决面和我日志完全对得上(
pickAttachableskip /rotateStalePredecessor排除无 health 候选 / 同车道前任只等 5s 就走 +1)。对「有界耐心 + 强制接管」的方向没有异议,只补三个实现细节,都会直接决定 10 分钟窗口是否成立:观察计数必须落盘,不能只在内存。
跨 bring-up 周期累计观察(首次观察时间戳 + 观察次数)只有在 state 目录(~/.local/state/billion-context/)持久化才有效。attach/launch 的判定在每个新客户端进程里做,而故障期间客户端恰恰在反复重启——我那单的日志里[dsh-client] proxy bootstrap failed每轮都重复出现。计数只在内存的话,每次重启清零,窗口永远拼不满。建议把「请求被本地拒绝」当作硬观察事件,短路耐心。 时间窗口只在定期 attach 时有采样点;
bili-native: oversized request refused locally是 holder 无用的直接证据(请求根本没发出去)。建议:出现一次 oversize refusal → 置proven_useless,剩余窗口压到 ≤60s,不必再等满 10 分钟丢掉更多轮次。给 BUSY 加个廉价的活/死判别,比纯时间更准。 两次采样 holder 的 CPU 时间(win32
GetProcessTimes/ POSIX/proc/pid/stat的utime+stime),5s 内增量 ≈0 说明它不在压缩而在卡死/挂起,可直接接管;增量明显才算合法工作,走耐心窗口。合法的多分钟压缩会持续吃 CPU;而饱和到答不出 1.5s 探针的实例通常既无进度也无 CPU。两个小点:
taskkill /F /T到端口真正可 bind 之间有 TIME_WAIT,建议 bind 重试 ~2s 再判code 1,否则接管成功也会误报 spawn 失败。实例 pid X 无健康应答,N 分钟后接管),否则这 10 分钟里用户只见 422、不知道在等什么。4/5/6 拆单没问题。
测试意愿:这台机器有完整复现环境与原始日志(
bili.log15:10:05–15:26:01:旧 holder pid 148832 占 18787 + fingerprint 不符、childexited (code 1)、恢复后 18788 粘滞)。分支/prerelease 出来后给我版本号,我可以原样跑一遍,重点验「旧 holder 被接管 → 18787 恢复可 bind → 同一大会话不再 422」。需要的话我把这段日志全文贴上来。ranxianglei commentedon Nov 8, 2026
(ai: qwen/qwen3.8-27b)
感谢复核——三条裁决面与你的日志逐条对上,说明根因定位没有跑偏。对你补的三个实现细节,裁决如下:
1. 观察计数落盘 —— 采纳,这是修复成立的必要条件。 你说得对:故障期间客户端反复重启(你日志里每轮
bootstrap failed),纯内存计数每轮清零,10 分钟窗口永远拼不满。改为 state 目录下的 JSON 账本(原子写、损坏即重建、按origin#pid键控、死 pid 自动清理),跨进程/跨重启累计观察。2. oversize refusal 短路耐心 —— 方向同意,拆单跟进。 它是一条独立机制:证据在客户端侧(
bili-native: oversized request refused locally),决策在启动器侧,需要设计跨进程信道(同一 state 目录的标记文件是候选方案)。按「一 issue 一机制」不混进本 PR;本版本保留 10 分钟上界,短路作为加速项单独跟踪。3. CPU 活/死判别 —— 方向同意,同样拆单。 win32
GetProcessTimes/ POSIX/proc/pid/stat采样有自己的误判面(I/O 停顿的合法压缩 vs 真卡死),误杀代价是丢掉在途回合,需要独立设计与证据,不进本 PR。两个小点:
MAX_LISTEN_ATTEMPTS=17),且本修复在 spawn 前会验证「TCP connect 已失败」才放行,双层覆盖。若你的复现里接管成功后仍见 code 1,把那段日志发我,看具体间隔再定要不要加显式退避。测试意愿收到:主修复走分支
2026-11-08_wedged-same-lane-takeover(PR 稍后贴链接)。prerelease 发布需要 owner 批准,我先给分支号 + commit sha,发布批准后补正式版本号。验收点就按你说的「旧 holder 被接管 → 18787 恢复可 bind → 同一大会话不再 422」。4/5/6 三个拆单在主 PR 提交后开,带来源标记。
TOSKA1226 commentedon Nov 8, 2026
分支名收到,我会盯
2026-11-08_wedged-same-lane-takeover和对应 PR。一条补充,关于账本键控:
按 origin#pid 键控 + 死 pid 自动清理漏掉了 pid 复用这个反向风险。Windows 上 pid 在密集 kill/重启后回收很快,如果旧账本键origin#148832被新进程复用,新实例一出生就带着「已观察 N 分钟」的历史,下一次观察直接进强制接管——误杀一个活着的实例,代价比现在的僵尸更直观(正在服务的 session 被打断)。建议键里带上进程创建时间(win32
GetProcessTimes.ftCreationTime/ POSIX/proc/pid/stat第 22 字段starttime),键 =origin#pid#starttime;创建时间不同即视为新实例、计数归零。这比「死 pid 清理」更严格,且同样是纯本地读,不额外要权限。另一个待确认项:TIME_WAIT 那层你提到
MAX_LISTEN_ATTEMPTS=17,这 17 轮的间隔是阶梯退避(10×500ms 那种)还是固定短间隔?如果总时长 < 4s,被杀进程遗留的 socket 在 Windows 上未必全部释放干净。我复现时会记录「kill → 首次 bind 成功」的真实耗时给你,省得靠猜。分支/PR 我这边现在还没看到(
gh api .../branches和gh pr list里都没有wedged-same-lane-takeover),出来后我立刻在这台机器上跑那三个验收点。ranxianglei commentedon Nov 8, 2026
(ai: qwen/qwen3.8-27b)
PR 已出:https://github.com/ranxianglei/billion-context/pull/5972(分支
2026-11-08_wedged-same-lane-takeover,提交33bec442d9,基于最新 master rebase 后重新验证)。PR 标签:bug / p1 / dsh / scope/wiring。你上一轮(m00359)的三个点,逐条回:
origin#pid#进程创建时间戳:linux 读/proc/<pid>/stat第 22 字段(starttime)、win32 用 powershellGet-Process StartTime、其他 POSIX 用ps -o lstart。创建时间不同 = 新实例,计数归零、窗口重开;戳取不到时降级为origin#pid(保守方向:宁可多等一轮,不会误杀)。测试里有专门的 pid-reuse 行验证「旧键留着历史、新键从 1 开始」。wait-predecessor= 每 500ms × 最多 10 次(共 5s),且仅当同车道前代进程还活着(isPidAlive)时才触发——我们硬杀之后 pid 已死,这条永远不会触发;任何残留 EADDRINUSE 会立即 ladder 到 +1(零延迟),正是我们修的那次漂移。所以真正的风险窗在「launcher 验证端口释放」与「子进程 bind」之间(约 spawn 耗时 100–500ms)。两层缓解:① 硬杀阶段成功后先等 2s(WEDGED_POST_KILL_SETTLE_MS)再放行 spawn(优雅/SIGTERM 路径监听器是正常关闭的,无残留,不等);② Node 默认 SO_REUSEADDR,连接型 TIME_WAIT 不挡 bind,挡不住的只有被杀 LISTENER 的残留释放。你复现时记的「kill → 首次 bind 成功」实测耗时对我很有价值,欢迎贴回来校准这 2s。修复摘要(机制层面):决策表补上缺失的一格——同车道、异字节、健康探测无应答(BUSY)的规范端口驻留者。跨 bring-up 周期累计观察(持久化账本,state dir 原子写——客户端每轮 bootstrap 失败都重启,内存计数永远填不满窗口;这是你 m00313 第一条意见,已采纳),连续 ≥10min(=2× watchdog 硬超时)且 ≥3 次独立观察后走受监督的强制接管:优雅 shutdown 请求(3s)→ 等监听关闭(5s,未关则 SIGTERM + 2s,POSIX)→ 仍占着则硬杀(SIGKILL / win32
taskkill /F /T)+ 5s → 验证监听器确实关闭才放行正常 spawn;任何一步失败都回落到原封未动的 spawn ladder,不会比今天更糟。耐心窗口开始时会打一条用户可见日志(不再只有 422);受害方至多丢当前 in-flight turn(会话逐轮持久化)。八问:
killAbandonedChild行为保持(组杀 + 句柄级 SIGKILL,ESRCH 静默与 master 一致)。唯一新增行为就是这张缺失格本身的处置,已在 PR 描述里按 old→new 列表披露。killAbandonedChild同款纪律([Bug] Plugin bootstrap spawns a new instance while the old one still holds the port and drops proxying (goes direct): la | [Bug] 插件 bootstrap 在旧实例仍占端口时新起实例并放弃代理(goes direct):大会话随即以未压缩历史撞上游上限(400 Range of input length / 413) #4125:失败必须可见,不得吞掉)。wedged-takeover.json(空时自动删除,损坏时大声告警并降级为重新计时,绝不阻塞 bring-up)。后续项(按平台政策 2026-11-01 不另开 issue,留此备查,owner 可挑任意一条立项):
判定:本地定向套件全绿(新增 16 测 + 相邻 launcher/port-owner 共 433 测 0 失败,1 个既有 skip)+ typecheck 干净;全量回归以 CI 为准。需人工 review——新增对仍在服务进程的强杀路径 = 产品决策,不合 auto-merge 条件。等你机器上的三验收点实测结果。
TOSKA1226 commentedon Nov 8, 2026
本机三验收点实测结果(0.1.213 / 分支
2026-11-08_wedged-same-lane-takeover@ 33bec44)环境:Windows 11,node v24.15.0;构建方式
npm --prefix kernel run build && npm --prefix pi-subagents run build && npx tsup(dist/index.js 5.75 MB);新增测试node --import tsx --test tests/issue5955-wedged-same-lane-takeover.test.ts→ 16 tests / 16 pass / 0 fail(含真实 powershell birthstamp 用例)。方法(全隔离,不碰线上代理):独立
XDG_STATE_HOME;假 holder = 只 accept 永不回应的 net server(正是 BUSY 形状),写proxy-origin(lane=dsh、codeFingerprint=sha256:deadbeef…、pid=holder);用processBirthStamp(holderPid)复刻账本 key,预置firstBusyAt = now-11min、observations=3(跳过 10 分钟耐心窗口),然后调ensureProxyRunning({host:"127.0.0.1", port, passthrough:false, debug:true, lane:"dsh"})。验收点 1:旧 holder 被接管 —— PASS
日志链(两轮一致,18790 试跑 + 18787 真端口):
timings(18787):forceAt 2030 ms、releasedAt 14463 ms、kill→release 12433 ms、kill→healthy 13788 ms;账本条目接管后已删除(读取为 null);holder 退出码 1。
验收点 2:18787 恢复可 bind —— PASS
接管后子进程成功绑回 18787 并 health 200(v0.1.213);清场后 18787 无 listener、可再次 bind。此前"zombie 占 18787 → 新进程 EADDRINUSE 退出 → 被迫落 18788"的死锁不再发生。
验收点 3:同一大会话不再 422 —— 触发条件已消除(含现场旁证)
422
bili_gate_oversize_refusal_*只在代理没起来时由src/agent/native-intercept.ts:708(no_live_state)与:879(fallback)产生;接管成功后 bring-up 返回健康 handle,这两条分支不再进入(接管全程日志无任何 oversize 拒绝)。现场旁证:本机一个 14746 msgs / 24.3 MiB 的会话当前每轮local=1.76–1.84s、[acp-usage] input=251519 cached=250624 (cache hit 100%),转发正常。kill → 首次 bind 成功耗时(校准
WEDGED_POST_KILL_SETTLE_MS=2000)taskkill /F /T后端口立即可 bind,4 次实测:226 / 162 / 191 / 252 ms(中位 ~191 ms)。当前 2000 ms settle 约为实测值的 8–10 倍,安全但每次接管多付 ~1.8 s。两个可优化点(实测暴露)
requestShutdown+waitForListenerClose(5s)+WEDGED_TERM_GRACE_MS(2s)三段必然空转,硬杀前这 ~12 s 是纯等待。建议对 BUSY 候选把优雅关闭阶段整体限流(例如单段 500 ms,或跳过 graceful 直接taskkill /F /T),最坏接管时间可压到 ~2.5 s。bili: direct SIGKILL of wedged holder <pid> failed (kill ESRCH) — its listener may survive the cleanup (#4125)。实际 taskkill 已收割进程,属噪音;建议 win32 分支在 taskkill 成功(或进程已不存在)时抑制该条。反向验证(空账本 / watch-only,端口 18791)
假 holder 存活 13 分钟期间:只记录 1 次观察
{pid:68380, firstBusyAt, lastObservedAt, observations:1},绝不杀;随后正常 ladder 到 18792 起新代理并 health 200。符合"证据不足不动手"的设计意图。(附注:我自己的 harness 退出时与 taskkill 竞争触发过
Assertion failed: !(handle->flags & UV_HANDLE_CLOSING), file src\\win\\async.c, line 76,属测试脚本问题,与产品代码无关。)fix(launcher): cap the BUSY takeover budget, stop reporting reaped pi…
fix(launcher): field-tune wedged-takeover timings + quiet win32 ESRCH…
ranxianglei commentedon Nov 8, 2026
(ai: qwen/qwen3.8-27b)
三个验收点全 PASS,这个实测非常关键——死锁单元确实在真实 Windows 环境里解掉了(18787 释放、子进程绑回、账本条目删除、422 触发条件消失)。你测出的 kill→bind 162–252ms 和 12.4s 接管耗时都直接派上了用场。两个优化点都已修掉并推到分支:commit
2826c518f8(已 rebase 到e7d717937c之上——那是另一会话按 CI 门禁把未使用的wedgedLedgerFilePathexport 去掉的一行改动,无行为变化)。① 12.4s 空转 → 收紧各段预算(采用「限流」而非「跳过 graceful」)
最坏释放判定从 ~15s 降到 ~6s(+2s kill 后 settle 不变)。保留 graceful 的理由:它成本现在封顶 ~3s,而万一 holder 的循环恰好在升级后短暂解锁,仍能拿到干净退出;对「永不应答」的 BUSY holder,每段只是 200ms 轮询间隔的紧倍数。若 reviewer 认为连这 ~3s 都不该付,「直接跳过 graceful 硬杀」是一行改动的事——留给人工评审拍板。另外
requestInstanceShutdown的参数化是可选参数,#3544 轮换路径保持原 3s 默认,行为零漂移。WEDGED_POST_KILL_SETTLE_MS=2000有意保持不动:你的实测中位 ~191ms,2s 是 ~8× 裕量;这条路径每个 wedge 事件只走一次,而设小了后果正是我们要修的 +1 漂移复发。② win32 ESRCH 误报 warn → 已抑制
你抓到的那条正是
hardKillProxyTreewin32 分支的回退路径:taskkill 已收割进程树后,directprocess.kill抛 ESRCH(进程已不存在),被我错误地当成「可能残留」告警——每次成功接管都刷一条假警报。现在 win32 与 POSIX 分支一致:ESRCH = 目标已达成,静默。成功接管后这条不会再出现。其他确认:你的反向验证(watch-only 13 分钟 → 恰好 1 次观察、绝不杀、正常 ladder 到 18792)正是「证据不足不动手」的预期行为,谢谢钉住;harness 里的
UV_HANDLE_CLOSING断言确实是测试脚本与 taskkill 竞争所致,与产品代码无关,同意不处理。行为变更披露(旧→新):仅上表四项接管阶段预算 + win32 ESRCH 告警抑制;耐心窗口(10min/≥3 次观察/5min 断裂重启)、账本格式、kill 语义均不变。PR 正文已同步更新("Field-test feedback incorporated" 一节)。
状态:PR #5972 新 head
2826c518f8CI 运行中(当前 0 failed / 11 pending);此 PR 含新 kill 路径 = 产品决策,需人工评审合并;prerelease 仍需 owner 批准。想复验时间线的话拉2826c518f8即可,预期最坏接管 ~6s+2s settle,且那条吓人 warn 消失。TOSKA1226 commentedon Nov 8, 2026
Confirming the timeline on my side and one data point for the "keep or skip graceful" call you're leaving to review.
Re-verified on 2826c51 (branch
tests-wedged-budget, parent = your head): 18/18 tests pass in 2.8s,tsc --noEmitexit 0,check-unused-exportsclean (the single-tokenexport function hardKillProxyTreekeeps the gate green because a test imports it),check-test-import-graphok. The two assertions are written so your envelope survives a re-tune: shutdown POST budget must be> 0 && < 3000, and total escalation staging must stay>= 2000 && <= 6000— a regression back to 3s+5s+2s+5s fails with the measured ms in the message, and a re-tune inside your new table stays green.On whether the ~3s graceful leg is worth paying: in both of my field takeovers the holder was BUSY by construction (TCP accepted,
/__bili/healthnever answered), and the graceful leg yielded exactly zero — 1000ms POST budget burned, then the full 2000ms post-shutdown wait, then SIGTERM no-op on win32, then the hard kill that always had to happen. That is not an argument to remove it, it's an argument that the decision to try is already made earlier than the POST: at escalation time the ledger hasobservations >= 3spanning>= 600s, i.e. the holder's loop has demonstrably failed to drain for ten minutes. So the cheap discriminator is not a new knob, it's the ledger itself — when escalating on a candidate withobservations >= WEDGED_MIN_OBSERVATIONS, the graceful leg cannot succeed by evidence, and skipping it costs one boolean. Keeping it is defensible as belt-and-braces (cost capped at ~3s), but if review wants it gone, gating on the existing observation count is a better shape than a human override flag: it preserves graceful for a first-observation escalation and drops it for exactly the ten-minute-dead case we measured.Also worth recording for whoever reviews the kill path:
WEDGED_POST_KILL_SETTLE_MS = 2000against a measured 162–252ms (median ~191ms) is right — 8–10× margin on a once-per-event path, and starving it only reinstates the +1 drift. No objection.Tests-only branch stays parked at
016d933; I'll rebase to master and open it against master (not a stacked base) once #5972 merges, so the core matrix actually runs instead of beingci:full-skipped.Merge pull request #5972 from ranxianglei/2026-11-08_wedged-same-lane…
TOSKA1226 commentedon Nov 9, 2026
合入确认
#5972已由 @ranxianglei 于 2026-11-09T01:26:21Z 合入 master,merge commit6d750efe(2 parent:c7589e92+2826c518,走 merge commit 而非 squash)——分支上的三笔提交(33bec442/e7d7179/2826c518)原样保留在 master 谱系里。字段实测数据驱动的四项预算(1s / 2s / 1s / 2s + 2s settle)与 win32 ESRCH 抑制现已生效。补测试:PR #5990
维护者侧的
2826c51是纯src/launcher.ts改动(+33/−14),没有带任何测试——这条路径的终点是对活进程taskkill /F /T,安全边界全靠代码自觉。因此补了tests/issue5955-wedged-same-lane-takeover.test.ts(18 tests,新文件),基于当前 mastere7b3d282rebase:#5990
覆盖面(每条都对应一个"如果错了会杀错或杀不掉"的分支):
code-fingerprint-mismatch才入列;健康旧版仍走 安装新版本后旧代理永远不换新:detach 代理躲过应用退出,重启只是重新挂上旧进程(「永远更新不了」) #3544 轮转、健康新版仍按 [Bug] Plugin bootstrap spawns a new instance while the old one still holds the port and drops proxying (goes direct): la | [Bug] 插件 bootstrap 在旧实例仍占端口时新起实例并放弃代理(goes direct):大会话随即以未压缩历史撞上游上限(400 Range of input length / 413) #4125 采纳、同 build BUSY 只 attach 绝不杀(dsh lane: BUSY-drop → EADDRINUSE detour → adopt-twin race is silently lost ~half the time, and a one-shot bootstrap failure then becomes permanent #4811)、user-zone / 外车道 / 异端口 / 显式 attach 一律不动。origin#pid#birthstamp键,回收后的 pid 不能继承前代的 BUSY 历史。(0, 3000)ms、整体升级落在[2000, 6000]ms —— 未来再调WEDGED_*数值仍然通过,但回退到 3s 优雅阶梯会失败,这是本次修复的回归护栏。本地:18/18 通过(跑 3 次)、
tsc --noEmitexit 0、check-unused-exportsclean、check-test-import-graphok。src 侧仅一处把hardKillProxyTree加export供 ESRCH 静默测试直接驱动,无行为变化。#5980已在其 base 分支合入后自动关闭,其 src 半边被2826c51覆盖,测试半边由 #5990 接管。fix(launcher): force-takeover the wedged same-lane proxy holding the …
Merge pull request #5989 from ranxianglei/2026-11-09_wedged-takeover-…
test(launcher): cover the ranxianglei#5955 wedged same-lane takeover …
test(launcher): cover the ranxianglei#5955 wedged same-lane takeover …
Merge upstream/master (1f311dd): adopt ranxianglei#5986 degenerate wh…
test(launcher): cover the ranxianglei#5955 wedged same-lane takeover …
TOSKA1226 commentedon Nov 9, 2026
#5989 landed the coverage (
2026-11-09_wedged-takeover-coverage, commit 016d933, author TOSKA1226) — the 18 cases are on master now, together with the one-wordexportonhardKillProxyTreethey need.One follow-up that came out of that merge: the new test file matches
SPAWN_REinscripts/run-ci-tests.mjs:103(a realspawnSync(node, ["-e", ""])to fetch an already-reaped pid forprocessBirthStamp, plus the mockedspawnImpl), so it must be listed inLIVE_FILES(scripts/live-files.mjs). Right now master has it unregistered, which makes the tripwire fail all 11test (ubuntu|windows, 22|24, 1..5)shards for any PR — master's own push checks only run the e2e lanes, so it stays green there and the first PR to touch the core matrix eats it.Filed as #5990 (2 files, +15/−1: the registration plus an assertion that tolerates the ledger's documented one-observation loss). Locally the gate replays clean:
spawners=60 registered=62 missing=[].test(launcher): cover the ranxianglei#5955 wedged same-lane takeover …
test(launcher): cover the ranxianglei#5955 wedged same-lane takeover …
Merge origin/master into fix-4566-lane-port-sticky (#5064 conflict sync)
Merge pull request #5990 from TOSKA1226/tests-wedged-takeover-ladder
merge: sync master into 2026-11-04_keepalive-timer-unref (live-files …
fix(launcher): widen the win32 birth-stamp powershell budget 10s → 20s (
fix(launcher,storage): #7156 fixes 1+2 — never spawn a second same-la…
fix(launcher): widen the win32 birth-stamp powershell budget to 30s