Skip to content

[PR] fix: give-up client event names the unavailability branch (#4125) - #4389

Closed
ranxianglei wants to merge 1 commit into
masterfrom
2026-10-23_4125-giveup-reason
Closed

ranxianglei wants to merge 1 commit into
masterfrom
2026-10-23_4125-giveup-reason

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

Related to #4125 (observability slice; does NOT close it — the root cause of the multi-instance / hours-long direct-send episode is still being triaged against reporter evidence).

What this PR does

The proxy respawn gave up — model traffic goes direct (uncompressed) dsh client event was one bare sentence with no branch information. A user stuck in a direct-send episode could not tell whether the proxy was dead, saturated, failing to spawn, or whether a send had failed with no replacement — exactly the self-diagnosis gap #4125's proposal 4 asks for.

Changes:

  1. src/agent/native-intercept.ts — the two send-failure recovery paths (routed /bili/ branch and raw-URL branch) now record state.lastUnavailableReason = "send-failed" immediately before firing onGiveUp. The resolver paths already recorded dead / busy-exhausted / bootstrap-failed; every give-up firing site now has a fresh reason at fire time. UnavailableReason gains the member. Existing consumers compare only against "busy-exhausted" (opencode-native refusal wording), so the new member is inert elsewhere.
  2. src/agent/dsh-native.ts — the three duplicated inline onGiveUp closures collapse into one fireGiveUp seam that appends : <reason> to the persisted client-event line when the interceptor recorded one, keeping the original bare wording when none is present. _resetRegisterForTest clears the reason between scenarios; _nativeStateForTest exposes the shared state to suites (follows the file's existing ForTest-hook pattern).
  3. src/launcher.ts — killAbandonedChild no longer swallows win32 taskkill /F /T failures or a failing final SIGKILL; both now log a warn naming the pid. A surviving listener after cleanup is exactly the confusion [Bug] Plugin bootstrap spawns a new instance while the old one still holds the port and drops proxying (goes direct): la | [Bug] 插件 bootstrap 在旧实例仍占端口时新起实例并放弃代理(goes direct):大会话随即以未压缩历史撞上游上限(400 Range of input length / 413) #4125's netstat showed (two processes on one port).
  4. tests/issue4125-giveup-reason.test.ts (new) — pins both halves of the contract: the interceptor writes the reason BEFORE firing give-up for all four branches (dead, busy-exhausted, bootstrap-failed without firing, send-failed through the full fetch chain with direct-send still happening), and the dsh seam appends the reason to the persisted line / keeps the bare wording when absent.

Behavior changes (intended)

  • The give-up client-event line gains a : <reason> suffix (dead | busy-exhausted | bootstrap-failed | send-failed) whenever the interceptor recorded why nothing was routable. Old value → new value: proxy respawn gave up — model traffic goes direct (uncompressed) → same sentence + : <reason>. Hosts parsing the exact old string will see the longer form; the prefix is unchanged.
  • Two new warn lines in bili.log when the win32 abandoned-child cleanup reports failure (previously silent). Log-only; no control-flow change.
  • No wire, config, persistence, kernel, or env-surface changes.

Verification

Local targeted suites green on this head:

  • tests/issue4125-giveup-reason.test.ts tests/dsh-native.test.ts tests/issue2187-dsh-spawn-retry.test.ts tests/issue3380-dsh-rereg-selflock.test.ts tests/issue2082-dsh-recovery.test.ts → 79 pass / 0 fail
  • tests/native-intercept.test.ts tests/native-intercept-selfheal.test.ts tests/issue3116-spawn-failure-diagnostics.test.ts tests/launcher.test.ts → 421 pass / 1 pre-existing skip / 0 fail
  • npm run typecheck clean

Full-suite verdict comes from CI on this head.

@github-actions

github-actions Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-23_4125-giveup-reason (b3d8ee7)

Latest build — install this exact version

npm install -g billion-context@0.0.5332-pr.4389.78396a8

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Recent builds (commit → version)

commit version build
b3d8ee7 billion-context@0.0.5332-pr.4389.78396a8 build
b23c007 billion-context@0.0.5322-pr.4389.53b1cab build
3eb439b billion-context@0.0.5316-pr.4389.eaa61a0 build

Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr4389.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重:不重复。 按失败签名(give-up 客户端事件缺分支信息 + win32 孤儿子进程清理静默吞错)检索了 open/closed 的 issue 与 PR,无其他条目做同一修复;相关但机制不同的条目(#3544 优雅轮换 / #3654 健康探测 busy 判定 / #3044 twin 冲突退出 / #2187 spawn 超时孤儿清理)已在 #4125 里交叉关联,与本 PR 不重叠。本 PR 是 #4125 建议 4「gave up 日志带原因」的落地 + 一处相邻小修,明确不关闭 #4125(根因仍在等报告者证据)。

标签: bug / dsh / scope/wiring / p3(理由:可观测性缺口类缺陷;用户可见面是 dsh 客户端事件行;属宿主车道接线层;纯诊断/日志改善、零路由行为变更——事故本身的 p1 记在 #4125 名下)。

这个 PR 是我前一个会话在 #4125 triage 中开的(见 #4125 floor 4),本轮没有重做,而是独立审了 diff 并在最新 master 上重新验证。

陈旧基线处理(本轮主要动作):分支落后 master 269 个提交,三个源文件自 base 以来都在 master 上被改过;且原 head(c13f64cf1)上 38 个 CI check 全部 cancelled——没有任何有效 CI 结论。处理:rebase 到当前 master tip(70177775c,无冲突,逐行 diff 确认改动内容与原提交完全一致),本地重跑 typecheck + 定向套件后,用 force-with-lease 把 PR head 更新为 3eb439b。CI 会在新 head 上跑全矩阵。

验证结果(新 head 3eb439b)

  • npm run typecheck clean
  • 定向套件(dsh 车道 5 文件 + 拦截器/launcher 4 文件):503 pass / 0 fail / 1 pre-existing skip
  • 全量结论以 CI 在新 head 上的结果为准

独立审核(逐项)

  • ① 方向关:正确。修的是真缺口——三个 give-up 触发点各自为政,其中 send 失败两条路径根本不写 lastUnavailableReason,dsh 侧三处重复闭包只透传固定文案。「先记录原因、再触发 give-up」的顺序契约打在机制层,不是表面补丁。
  • ② 回归清点(diff 触碰路径承担的既有行为,逐项过):
    • give-up 语义(register.base 清空 + giveUpSeq++):三处原样保留 ✅
    • 无原因时的输出:保留旧句逐字节一致(seam 不硬造原因)✅
    • opencode-native 拒答措辞:只比较 === "busy-exhausted"(src/agent/opencode-native.ts:138/:207),undefined 与新成员 "send-failed" 落同一通用分支——PR 声称「新枚举成员对他惰性」成立 ✅
    • 全仓只有 dsh 持久化这条客户端事件行(该字符串在 src/ 中仅 seam 一处;opencode.ts:89 的 onGiveUp 只删环境变量)——范围完整无遗漏 ✅
    • master 自分支点后未新增 onGiveUp 闭包(两棵树都恰好 3 个赋值点)——「三闭包收敛」在 rebase 后依然完整 ✅
    • send-failed 路径仍然照发直连(测试断言 routed+direct 两次尝试都发生)✅
    • killAbandonedChild:控制流零变化,仅把 win32 taskkill/SIGKILL 失败从静默吞掉改为 warn(带 pid)——高危模式「fail-fast 变吞错」在这里是反方向(吞错变上报)✅
    • wire/config/持久化/kernel/env:零触碰(diff 仅 4 文件)✅
  • ③ 代码关:seam 在触发时从共享 state 读 reason(不是闭包构造期捕获),正确避免陈旧值;_resetRegisterForTest 同步清 reason 防跨场景污染;_nativeStateForTest 沿用该文件既有 ForTest-hook 模式。命名、注释风格符合仓库惯例。
  • ④ 测试有效性:断言有牙——拦截器测试在 onGiveUp 回调内部捕获 lastUnavailableReason,直接证明「先写后触发」的顺序(顺序回归则捕获值必错);四分支全覆盖含负例(bootstrap-failed 只记录不触发);send-failed 走真实 installNativeFetchIntercept fetch 链(非 mock 内部件),deepEqual 钉住两次请求的精确顺序(routed 后 direct);dsh seam 用 $ 锚定完整行形(含无原因裸句)。死端口 bind-then-release(fix: emit /acp panel as persistent custom message and strip it from the proxy model context (#359) #360 纪律)、hermetic XDG_STATE_HOME、ambient proxy env 密封——确定性,不赌环境运气。
  • ⑤ diff 整洁度:4 文件、1 commit、每行都服务于目的,无空白/锁文件/生成物 churn。
  • 行为变更披露(PR body 已声明,我核实无误):give-up 行在有原因时追加 : <reason> 后缀(dead|busy-exhausted|bootstrap-failed|send-failed),前缀不变、精确解析旧串的宿主会看到加长形式;两条 Windows-only 新 warn 日志行。无其他漂移。

按八问汇报

  1. 是什么问题:直连期间宿主侧唯一现场线索是一句不带分支信息的 give-up 事件,用户无法自查卡在哪个分支(代理死了/饱和/起不来/发送失败无替补);另 Windows 上两处清理失败被静默吞掉,残留监听器不留痕迹。
  2. 什么原因导致:dsh 三处 give-up 闭包各自只透传固定文案;拦截器 send 失败恢复路径从不写 lastUnavailableReason;killAbandonedChild 对 taskkill/SIGKILL 错误 catch 后丢弃。
  3. 影响范围与严重程度:只影响诊断能力(所有直连 episode;持久化该客户端事件行的只有 dsh 车道);排障体验缺陷,路由/wire 行为零变化。
  4. 怎么修复的:统一「先记录原因、再触发 give-up」顺序契约(四分支在触发时刻都有新鲜原因);dsh 三闭包收敛为一个 seam,有原因才加后缀、否则保持旧句;清理失败由静默改为 warn(带 pid)。
  5. 是否有回归问题:未发现——见 ② 清单(give-up 语义、裸句输出、opencode 措辞惰性、直连保持、launcher 控制流、无阈值/时序漂移,均逐项核实)。
  6. 是否破坏缓存:不涉及(纯客户端事件通道 + 日志行)。
  7. 是否改了设计方向:不涉及。
  8. 是否引入新配置:无。

判定:可合 —— 待 CI 在新 head 3eb439b 上转绿(本地 typecheck + 定向套件已全绿,rebase 无冲突、patch 内容不变)。合并权在人:#4389 。#4125 根因仍等报告者证据,本 PR 不关闭它。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

CI 结果更新(新 head):全矩阵已跑完——除 unused-exports 一条腿外全部绿。那条红不是本 PR 引入的:我在纯净 master(70177775c)本地复现了完全相同的失败——3 个 #4434(split-brain surface,合在 master tip)留下的模块内类型导出没人跨文件用:src/served-markers.ts 的 SplitBrainContest / ServedMarkerOptions、src/storage/runtime-status.ts 的 SplitBrainIdentity。本 PR 不碰这两个文件。

处置:按「CI 红几行小修顺手修掉」的纪律,已在本分支补一个独立标注 commit b23c007(去掉这 3 个类型的行首 export,checker 对 INTERNAL_ONLY 的处方;本地跑 node scripts/check-unused-exports.mjs 已过 + typecheck 仍干净)。与在途的 #4419(修同门另一组 corrupt-index-child 导出 + lock-sync)不同文件、无重叠,已在 #4419 里留言说明。当前 head:b23c007,等这条腿转绿即可合。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

CI 再更新(更正上一条 + 新发现一个 master 红):

  1. 更正:我此前说上一个 head「除 unused-exports 外全绿」——重新逐条核对 check-runs 后发现不对:那个 head 的 9 个 test shard 其实全是 cancelled/skipped(被后续 push 顶掉,从未跑完),当时没有有效的测试结论。抱歉给了不准确的状态。
  2. unused-exports 已确认转绿(b23c0079a 上该 job success,修复生效)。
  3. 但 9 个 test shard 现在全部在 preflight 就挂(未跑任何测试):scripts/run-ci-tests.mjs 的 SPAWN_RE 三保险报 tests/issue-split-brain-surface.test.ts 没登记进 scripts/live-files.mjs 的 LIVE_FILES。本地在纯净 master(70177775c)复现了同样失败——又是 master 红,[PR] Fix: False-positive warnings and true conflict visibility for multi-lane shared storage (automatic dead-process cleanup | 修复:多车道共用存储的误报警告与真冲突可见性(死进程自动清除/跨车道降级/脑裂横幅+一键关闭) #4434(split-brain surface)加了会 spawn 进程的测试文件却漏了登记。

处置(同前,CI 红几行小修顺手修):按 tripwire 要求的 #1248 并发假设审查过了这个文件——它只 spawn 一个不占端口的 sleep 子进程 + 一个 port: 0 临时端口的 proxy(makeOpts),每个测试 mkdtemp 隔离 dir/env、finally 全量恢复,--test-concurrency 4 下安全;按 append-only 槽位约定追加到 LIVE_FILES 末尾(slot=index 决定 zone 端口带)。commit 87e5129。本地验证:tripwire 逻辑逐字复跑干净 + 该套件单独跑 5/5 过。

当前 head:87e5129,等本轮 CI(这次 test shard 会真正跑起来)。

The 'proxy respawn gave up' dsh client event was a single bare sentence
with no branch information, so users hitting hours-long direct-send
episodes could not self-diagnose which failure mode they were in.

- native-intercept: the two send-failure recovery paths now record
  lastUnavailableReason='send-failed' immediately before firing onGiveUp
  (the resolver paths already recorded dead/busy-exhausted/bootstrap-
  failed); UnavailableReason gains the member. Consumers compare only
  against 'busy-exhausted', so the new member is inert elsewhere.
- dsh-native: the three duplicated onGiveUp closures collapse into one
  fireGiveUp seam that appends ': <reason>' to the persisted line when
  the interceptor recorded one, keeping the original bare wording when
  none is present. _resetRegisterForTest clears the reason between
  scenarios; _nativeStateForTest exposes the shared state to suites.
- launcher: killAbandonedChild no longer swallows win32 taskkill
  failures or a failing final SIGKILL — both now log a warn naming the
  pid, since a surviving listener is exactly the confusion #4125's
  netstat showed.

Behavior change (intended): the give-up line gains a ': <reason>' suffix
(dead | busy-exhausted | bootstrap-failed | send-failed) whenever the
interceptor recorded why nothing was routable; hosts parsing the exact
old string will see the longer form. No wire, config, persistence or
kernel changes.
@ranxianglei
ranxianglei force-pushed the 2026-10-23_4125-giveup-reason branch from 87e5129 to b3d8ee7 Compare October 23, 2026 17:17
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

冲突已解决,PR head 更新为 b3d8ee7f0(force-with-lease,旧 head 87e512959)。

本轮发现的关键事实:master 自分支点后合入的 7 个提交里,有两个正是本分支上两个 CI 止血提交的同款修复——#4441 的 2cb0347e4 与本分支 b23c0079a 补丁逐字节相同(去掉 served-markers/runtime-status 三个内部类型导出);#4440 的 dc380bd15 与本分支 87e512959 是同一处 LIVE_FILES 登记(仅注释措辞不同)。冲突就出在后者:scripts/live-files.mjs 同一位置双方各加了一版注释。

处置:rebase 到 master tip(fd973ca7d)——b23c0079a 被 git 自动识别为已应用而跳过;87e512959 冲突按「master 已合入的版本为准」解决(保留 #4440 的注释措辞),该提交遂变空并丢弃。结果:PR 现在只承载自己的修复,单 commit b3d8ee7f0(native-intercept / dsh-native / launcher / 新测试 4 文件),与 master 的 diff 和原 3eb439b2c patch-id 完全一致(f72c561a…)——rebase 零语义改动。

验证(新 head):npm run typecheck clean;定向套件(dsh 车道 5 文件 + 拦截器/launcher 4 文件)503 pass / 0 fail / 1 pre-existing skip。全量结论以 CI 在新 head 上的结果为准。

按八问汇报

  1. 是什么问题:系统转发的上游 PR 冲突——分支落后 master 7 个提交且 merge state dirty;深层原因是本分支的两个 CI 止血提交与随后合入 master 的 The unused-exports check on master is red: #4434 introduced 3 exported types that are only used within their own file, b | master 的 unused-exports 检查红了:#4434 引入 3 个只在文件内部用的导出类型,挡住所有基于 master 的 PR #4440/[PR] fix(#4440): remove exports of file-internal types from 3 files, restore unused-exports gate to green | fix(#4440): 删掉 3 个文件内部类型的 export,恢复 unused-exports 门为绿 #4441 撞了同一处修改。
  2. 什么原因导致:开 PR 时 master 红(unused-exports gate + LIVE_FILES 漏登记,均为 [PR] Fix: False-positive warnings and true conflict visibility for multi-lane shared storage (automatic dead-process cleanup | 修复:多车道共用存储的误报警告与真冲突可见性(死进程自动清除/跨车道降级/脑裂横幅+一键关闭) #4434 遗留),本分支顺手止血;期间这两处修复经独立 PR(The unused-exports check on master is red: #4434 introduced 3 exported types that are only used within their own file, b | master 的 unused-exports 检查红了:#4434 引入 3 个只在文件内部用的导出类型,挡住所有基于 master 的 PR #4440/[PR] fix(#4440): remove exports of file-internal types from 3 files, restore unused-exports gate to green | fix(#4440): 删掉 3 个文件内部类型的 export,恢复 unused-exports 门为绿 #4441)先一步合入 master,形成双写。
  3. 影响范围与严重程度:仅 PR 分支自身;对用户/运行时行为零影响。冲突面只有 scripts/live-files.mjs 一行注释;主修复 4 文件在 master 侧无人触碰,无冲突。
  4. 怎么修复的:rebase 到 master tip;逐字节相同的止血提交交给 git 自动跳过,措辞不同的那处保留 master 版本并丢弃冗余提交——收敛为「一个 PR 只带自己的闭环」,两笔止血的所有权归回 The unused-exports check on master is red: #4434 introduced 3 exported types that are only used within their own file, b | master 的 unused-exports 检查红了:#4434 引入 3 个只在文件内部用的导出类型,挡住所有基于 master 的 PR #4440/[PR] fix(#4440): remove exports of file-internal types from 3 files, restore unused-exports gate to green | fix(#4440): 删掉 3 个文件内部类型的 export,恢复 unused-exports 门为绿 #4441。
  5. 是否有回归问题:无——新 head 相对 master 的 diff 与原修复 patch-id 一致,未引入任何语义漂移;丢弃的两笔提交内容已逐字存在于 master。
  6. 是否破坏缓存:不涉及。
  7. 是否改了设计方向:不涉及。
  8. 是否引入新配置:无。

判定:可合 —— 待 CI 在新 head b3d8ee7f0 转绿(本地 typecheck + 定向套件已全绿)。合并权在人:#4389 。#4125 根因仍等报告者证据,本 PR 不关闭它。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

分支状态更新(owner 已接手收尾):分支被 force-rebase 到新 master,当前 head b3d8ee7(原 PR 单提交重放)。我此前推的两个修复 commit(b23c0079a 去 3 个内部类型导出、87e512959 登记 LIVE_FILES)已被合入 master 的 #4441(修 #4440,内容相同,另一会话并行立的项)取代——不在分支上了,但全部效力已在 master。

核对过:本 PR 的 4 个文件在我审过的 head 与 b3d8ee7 之间逐字节一致,前面的审核结论原样有效。

CI(b3d8ee7f0):11 条 workflow 已 10 绿(全部 e2e 车道 + build artifact + Windows codex overlay),只剩主 CI 矩阵(test shards + 各 gate,含 unused-exports / LIVE_FILES 三保险)还在排队等 runner(队列被其他分支的 run 占着)。跑完我再报最终结论。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

CI 最终结论(head b3d8ee7):全绿。 49 条 check-run:10 个 test shard(ubuntu 22/24 ×3 + windows ×5,#4439 权重分片后的新布局)全过;全部 gate 过——含此前红的 unused-exports 和 LIVE_FILES 三保险(#4434 遗留的两个 master 红已由合入 master 的 #4441 修复);static-gates 双平台过;e2e 车道按门控 skip。

至此本 PR 的审核闭环:方向(#4125 提案 4 的可观测切片)、回归面(give-up 全部触发点、reason 消费方、事件行唯一出处)、测试有效性(四分支 reason 时序 + dsh seam 双向断言)、diff 整洁度均已在前面评论里逐项给过结论;本 PR 的 4 个文件在审过的 head 与 b3d8ee7 之间逐字节一致。判定:可合,等你点合并。

@ranxianglei

Copy link
Copy Markdown
Owner Author

(ai: zhipuai-lb/glm-5.3)

已并入合并列车 #4551(master 合并,merge commit 保留原作者署名)。本 PR 关单收尾;回滚点为 #4551 单提交 revert。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working dsh p3 scope/wiring

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant