Skip to content

[PR] fix(hermes): adopt own spawned child via registry when the shared proxy-origin pointer is clobbered - #4658

Closed
ranxianglei wants to merge 15 commits into
masterfrom
2026-10-25_fix-hermes-spawn-clobber
Closed

ranxianglei wants to merge 15 commits into
masterfrom
2026-10-25_fix-hermes-spawn-clobber

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

What & why

The Hermes plugin (hermes-plugin/__init__.py) spawns its own bili proxy, then polls for it to become healthy. That poll only trusted the single machine-wide proxy-origin pointer and required its launchToken to equal this round's token.

That pointer is last-writer-wins. A concurrent bring-up (another lane starting, a second starter) that registers after our child overwrites it with a foreign token, so the launchToken match starves for the whole startup budget (SPAWN_WAIT_S). At timeout the loop calls child.terminate() — killing our healthy child — and the host goes inert: model traffic goes direct (uncompressed). That is exactly the failure reported in #4656. Our healthy instance was correctly registered in its own per-instance marker (instances/<id>.json) all along; the poll simply never looked there.

Fix

Mirror the TypeScript launcher fix already shipped for this exact mechanism (#2954). After the shared-pointer check, scan the per-instance registry markers for our child by pid and, once a health probe of the origin it recorded passes, adopt it:

  • Identity is by pid, not launchToken, because the registry marker deliberately omits the launcher-private token (src/server.ts strips it); a pid we just spawned cannot be faked by a concurrent writer.
  • Placed before the child-exit check, so a healthy self-registration wins even if the child process has already exited by that poll.
  • Scope-limited as requested: no new agent, no budget change, no storage or config change.

Behavior change (intended, flagged)

  • Old: when proxy-origin was clobbered by a concurrent bring-up, the Hermes plugin timed out, terminated its healthy child, and stayed inert (uncompressed direct traffic) until a manual restart.
  • New: it adopts its own healthy child via the registry and compression stays active. Restores the intended behavior; no other code path is touched.

Tests

  • spawn-clobber (new): end-to-end repro — pre-seeds a foreign clobbered pointer, the spawned child self-registers by pid; asserts the plugin becomes active (tools registered) with mode="spawn". Fails pre-fix (times out / stays inert), passes post-fix — verified by temporarily disabling the fix line.
  • _find_spawned_in_registry (new): pins the own-pid-only acceptance — a live foreign instance carrying its own launchToken and a stale dead-pid marker are both rejected; no child pid → never match. Deterministic (no subprocess).

Full tests/hermes-native.test.ts green on the rebased head (40/40); typecheck clean.

Closes #4656

…s clobbered (#4656)

The Hermes plugin's ensure_origin spawn-wait loop only trusted the single
machine-wide proxy-origin pointer and required its launchToken to equal this
round's token. A concurrent bring-up registering AFTER our child overwrites
that last-writer-wins pointer with a foreign token, so the match starves for
the whole startup budget and the loop SIGKILLs our healthy child at timeout
-> the host goes inert (model traffic direct, uncompressed).

Mirror the TypeScript launcher fix (#2954): after the shared-pointer check,
scan the per-instance registry markers for OUR child by pid (the registry
omits the launcher-private token and is keyed per instance, so it cannot be
clobbered) and adopt its origin once a health probe passes. It runs before the
child-exit check so a healthy self-registration wins even if the child process
has already exited by this poll.

Regression coverage: the spawn-clobber scenario reproduces the clobber
end-to-end and fails pre-fix; the _find_spawned_in_registry unit test pins the
own-pid-only acceptance (a live foreign instance with its own launchToken and a
stale dead-pid marker are both rejected).
@ranxianglei ranxianglei changed the title fix(hermes): adopt own spawned child via registry when the shared proxy-origin pointer is clobbered [PR] fix(hermes): adopt own spawned child via registry when the shared proxy-origin pointer is clobbered Oct 25, 2026
@github-actions

github-actions Bot commented Oct 25, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-25_fix-hermes-spawn-clobber (6d041a9)

Latest build — install this exact version

npm install -g billion-context@0.0.5973-pr.4658.8f5f618

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Recent builds (commit → version)

commit version build
6d041a9 billion-context@0.0.5973-pr.4658.8f5f618 build
a5590b2 billion-context@0.0.5935-pr.4658.0b526c4 build
f93a06b billion-context@0.0.5706-pr.4658.c5a45fa build
bc8a723 billion-context@0.0.5704-pr.4658.2ca9acc build

Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr4658.tgz
npm install -g package

Install instructions are refreshed on each push.

… share the spawned pid (#4658 review)

Review follow-up on the #4656 adoption helper: _find_spawned_in_registry returned the first pid-matching marker in filename order. If a stale marker from a previous run whose dead pid the OS later reused for our child sorts before our own marker (and its recorded origin happens to still answer healthy probes), the first-match rule would adopt the foreign origin. Among same-pid matches the newest started_at now wins — our child's own marker always postdates any stale one. The TS twin (#2954, src/launcher.ts) closes the same corner with a responder-pid check from /__bili/health; this keeps the two lanes aligned in spirit without adding an extra HTTP probe per poll. Test pins the tie-break with a dup-pid sub-case where the stale marker sorts alphabetically first.
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm/qwen3.8-27b)

查重: 同症状同机制的竞争 PR 存在——#4657(外部贡献)。两者都修「共享 proxy-origin 指针被并发启动覆写后认不到自己刚拉起的健康子进程」。机制比对:#4657 要求每个候选(共享指针 + 全部 registry 条目)launch_token == 本轮 token 且存活且过探测,但真实 registry marker 刻意省略 launcher 私有的 launchToken(src/instance.ts RegistryEntry 注释写明 "minus the launcher-private launchToken"),所以该条件对真实的自注册 marker 永远不成立——生产环境修不动,它的测试通过只是因为 fixture 往 registry 条目里写了 token。本 PR 的 pid 身份方案与真实数据形态吻合,且镜像已上线的 #2954。建议合入本 PR,#4657 被其取代(是否关闭由你定,agent 不关)。

标签: bug, hermes, scope/wiring, p1(并发拉起时 Hermes 宿主整体 inert 的 P1 可用性缺陷;hermes 客户端;协议接线层)——保留既有 pr 标签。


结构化八问

  1. 是什么问题:Hermes 插件自拉 bili proxy 后只信机器级共享指针 proxy-origin(last-writer-wins)+ 本轮 launchToken。并发启动在我们子进程之后注册、用外来 token 覆写指针,轮询整个预算期认不到自己的健康子进程,超时后 terminate() 杀掉它,宿主 inert、流量直连不压缩([P1][Hermes] Healthy spawned proxy is missed after proxy-origin is overwritten, disabling bili for the host process #4656 报的就是这个)。
  2. 什么原因导致的:轮询只认一个身份载体(共享指针+私有 token),而该载体可被并发写者覆写;子进程其实一直正确写着自己的每实例 registry marker(instances/<id>.json,刻意不含私有 token),但轮询从不看那里。与 TS launcher 侧 [Kimi Native] 多 Provider 切换旁路绕过、Thinking 丢失、上下文窗口失真及端口冲突问题反馈 #2954 同一机制,Python 插件侧此前漏修。
  3. 影响范围与严重程度:p1(可用性)。触发条件:≥2 个 lane/starter 并发拉起且注册顺序落在我们子进程之后。后果:Hermes 宿主完全 inert(直连不压缩)直到手动重启。单 lane 用户不受影响。
  4. 怎么修复的:镜像 [Kimi Native] 多 Provider 切换旁路绕过、Thinking 丢失、上下文窗口失真及端口冲突问题反馈 #2954——轮询循环在共享指针检查之后、child-exit 检查之前,按 pid 在 registry marker 里找自己的孩子;其记录的 origin 通过健康探测即采纳(mode="spawn")。身份用 pid 而非 launchToken(registry 刻意省略私有 token;我们刚 spawn 的 pid 无法被并发写者伪造)。范围受限:无新 agent、无预算变更、无存储/配置变更。
  5. 是否有回归问题:对 ensure_origin 既有行为逐项清点——① ATTACH_ENV 显式 attach 未动;② discover_instance lane-aware attach 未动(仍第一优先);③ starting-marker 仲裁未动;④ 自 token 共享指针采纳首分支字节相同;⑤ child 先于健康退出→None 路径未动(probe 门:死 child 的 origin 过不了探测,诚实落空);⑥ 超时 terminate+None 未动(clobber 场景提前获救是唯一差异);⑦ finally 清 marker 未动(测试断言 marker_left=false)。无语义漂移/吞错/阈值漂移高危模式。kernel/wire/config 均未触碰。
  6. 是否破坏缓存:不涉及(宿主侧路由决策,不触及缓存/重试/游标/去重键)。
  7. 是否改了设计方向:未改——这是已上线机制([Kimi Native] 多 Provider 切换旁路绕过、Thinking 丢失、上下文窗口失真及端口冲突问题反馈 #2954)的 Python twin,消息身份与压缩管线不动。
  8. 是否引入了新的配置:无。

行为变更(PR 已声明,复核确认):旧=clobber 场景下耗尽预算→杀健康子进程→inert 直到手动重启;新=经 registry 采纳自己的健康子进程,压缩保持活跃。属恢复设计意图,其他代码路径不受影响。

审核发现并已直接修(小问题直接修):_find_spawned_in_registry 原实现按文件名序返回第一个 pid 匹配项。角落情形:上一轮运行留下的 stale marker(dead pid 被 OS 复用给我们的 child)、文件名排序在我们自己 marker 之前、且其记录的 origin 恰好仍能过健康探测——三者同时成立时会误采纳外来 origin。改为同 pid 多匹配时取 started_at 最新者(我们的 marker 必然晚于任何 stale marker),并加测试子例钉住 tie-break(stale marker 字母序排前仍必须选新的)。已作为独立标注 commit f93a06b1f push 到 PR 分支(TS twin 用 /__bili/health 的 responder-pid 检查关同一角落;此处保持同等防护意图,不给每次 poll 多加一次 HTTP 探测)。

验证证据:

  • PR head 全量 tests/hermes-native.test.ts:40/40 通过(~22s,与 PR 声明一致);
  • 前置失败验证:换回 master 的 hermes-plugin/__init__.py 单跑新增 spawn-clobber 用例 → FAIL(AssertionError),恢复后通过——回归测试有牙;
  • CI(新 head f93a06b1f):全部静态门禁绿(build-artifact、gate×7、kernel-guard、lane-guard、lifecycle-ledger-guard、mysql-contract、pi-subagents-guard、version-guard、lock-sync、docs-*、config-surface-guard、test-import-graph、unused-exports、plan);11 个 test 分片仍在跑,绿了即完整闭环。

判定:可合(待 test 分片全绿;本地定向验证 + 前置失败复现均已完成,竞争 PR #4657 建议关闭)。

@ranxianglei ranxianglei added bug Something isn't working hermes 改动范围:内核之外的主机侧接线(src/agent 适配器、server 路由、注入拦截)/范围兜底/客户端标签 p1 scope/wiring labels Oct 25, 2026
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm/qwen3.8-27b)

CI 跟进:新 head f93a06b1f(含审核小修 commit)全量绿——test 全部 11 个分片(ubuntu/windows × node22/24)+ 各 e2e 车道(e2e-fake-codex / e2e-native-dsh / e2e-native-opencode / e2e-native-pi / e2e-codex-persona / e2e-pi-codex-ws / e2e-local-registry / Windows codex overlay)均 success。此前回复中「待 test 分片全绿」的条件已满足,判定维持:可合。

ci-check and others added 10 commits October 25, 2026 23:02
…ment

#4316 PR3/3 (restampRefsToLedger) re-homes citation numbers once on every
full-mode cold load; goldens pinning exact ref numbers are born-red. Re-pin
the 7 storage-lane tests (engine-proofs x6 + orphan-recovery x1) modulo that
renumbering — zero production code touched.

- golden-storage-drive.ts: per-turn citation-normalized digest (shaCiteNorm)
  and state digest modulo stripOneTimeAlignment (stateDigestAligned); the
  deep-sort helper is hoisted into golden-canonical.ts for sharing.
- corrupt-index-child.ts: new stripOneTimeAlignment normalizer — strips the
  refLedgerAligned stamp and re-homes refs onto canonical raw-id order so a
  clean full-mode read and the legacy fallback load compare equal.
- storage-engine-proofs.test.ts: wire legs assert shaCiteNorm equality vs
  the legacy restart leg and the committed goldens; state legs assert
  stateDigestAligned; rollback leg now spells BILI_STORAGE_UNIFIED=legacy
  explicitly (#3818 flip: absent resolves to full, deleting the env would
  vacate the legacy path).
- storage-orphan-recovery.test.ts: pins the content-addressable
  byRaw->byRef round-trip instead of a counter-era m00003 literal.
- golden/storage-render/*.json regenerated with shaCiteNorm/stateDigestAligned.

Converges the master-red storage-lane family (#4599 plan A, owner directive).
…dger parity masks

The #4316 one-time renumbering restamped messageRefs but left
state.tokenSnapshot keyed by the OLD refs, so post-restart bodies
carried stale per-message token counts (1.3K vs 1.4K) and the
full-mode-restart wire drifted off the legacy baseline (#4599).

persist.ts: two-pass remap (stage moves, delete old keys, then write
new keys) so entries whose new ref collides with another move's old
ref survive; fires only when superseded refs exist.

Aligned normalizer (corrupt-index-child.ts): the cacheLedger's
segment bookkeeping (be/bl/bs, unfinalized tail line), the derived
agg block, and per-line miss-attribution (prov/seam) are restart
experience, not content — no-restart and restart legs legitimately
differ there. Masked in the aligned view only; raw digests keep
full accounting. Goldens regenerated (stateDigest + aligned).

storage-engine-proofs 12/12, golden-storage-render 5/5 locally.
…persisted state (#4599)

The fork-survival test asserted immediately after the compress tool
response, but the fold lands in the live session state asynchronously
(#3573 replay semantics). Slow CI runners persisted the pre-fold state
('0 block(s) (compressions=0)' cold load) and the deepEqual diverged.
Bounded 5s wait on the in-memory fold; fast machines exit on the first
iteration. Product semantics unchanged.
The #4316 cold-load alignment reads the unified sidecar; its ingest of
just-flushed records completes asynchronously. Slow runners compared
the pre-ingest state and saw refLedgerAligned missing while the fold
itself had landed. Poll (bounded 5s) until the loaded state carries
the flag; fast machines exit on the first probe.
Git-for-Windows autocrlf checks the bundled .md posts out with CRLF;
the exact-match '---' fences then failed to parse and every post lost
its title/excerpt/date frontmatter (Windows CI: 'bundled meet-bili post
found' assertion). Normalize CRLF to LF inside parseFrontmatter when
the fence is CRLF-shaped; LF content is byte-identical as before.
Regression test with a CRLF fixture.
… storage mode (#4599)

The #4316 cold-load alignment sets refLedgerAligned only when the
unified store is available; config-clean machines (CI runners) run
legacy mode where it is an honest skip. The test asserted the flag
unconditionally, so every shard carrying this file was born-red on CI
while unified-config machines passed. Expectation now mirrors
store.replicasAvailable(); verified 49/49 in BOTH modes locally.
The ci-check lane's expectation change (c496cfb) exposed a real Windows
bug: stripInheritedProxy builds inlineEnv as a spread copy of process.env,
which keeps the original key casing (ComSpec) — Windows env vars are
case-insensitive but plain objects are not. planClientSpawn's exact
env.COMSPEC probe then missed it and silently degraded every wrapped spawn
to the bare-name 'cmd.exe' fallback (verified by repro on a real win32 box:
childEnv.COMSPEC=undefined while process.env.COMSPEC was set).

Fix: probe COMSPEC/ComSpec/comspec in planClientSpawn and planDshSpawn.
Verified on real Windows (Node 22): launcher-opencode-inline + fix-679 +
dsh-refresh 37/37; host tsc clean, 35 pass / 2 win32-skips.
@yanhuojiyusheng

Copy link
Copy Markdown
Contributor

补一条真实环境现场,支持按 pid 识别的方向。

本机 2026-10-25(CST):三个代理分别在 46291 / 45507 / 48383 监听成功,约一分钟后插件仍报 proxy did not become healthy within its startup budget,/acp 随后显示启动失败、模型流量直连;重启宿主后恢复。日志里「子进程健康」与「插件判死」并存,正是共享指针被并发方覆写后 launchToken 匹配饿死的形状。数据库大小可能扩大并发时间窗,但不是已确认根因。

一条边界说明,避免误导后来者:#4657 按 launchToken 扫描 registry 的补丁在生产环境是空操作(registry 条目本身不含该 token,见 src/instance.ts:467-485),所以「装了补丁后恢复」不能当作它的有效性证据,恢复更可能来自重启本身。判定仍应以 pid 路径的定向用例为准。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm/qwen3.8-27b)

感谢补充现场证据。三方并发拉起的场景与 #4656 的触发条件完全吻合:共享 proxy-origin 指针是机器级 last-writer-wins,后注册的一方用外来 token 覆写它,轮询整个预算期 launchToken 匹配饿死,超时杀掉自己健康的子进程 → 宿主 inert 直到手动重启。「子进程健康」与「插件判死」在日志里并存,正是这条路径的指纹(重启能恢复也符合:新一轮 token、单 lane 拉起时无并发写者)。

边界说明与审核结论一致:真实 registry marker 不含 launch_token(src/instance.ts RegistryEntry 刻意省略 launcher 私有 token),#4657 的 token 门对真实自注册条目永远不成立,「装了补丁后恢复」确实不构成其有效性证据——恢复来自重启本身。这也是审核建议采纳 #4658 的 pid 识别路径、并以定向 spawn-clobber 用例为准(前置失败复现 FAIL / 修复后 PASS 均已验证)的原因。

当前状态:#4658 CI 全绿(含一条审核跟进 commit f93a06b1f:同 pid 多 marker 时取最新注册),判定可合;#4657 被其取代,是否关闭由 owner 定。

ci-check and others added 2 commits October 26, 2026 09:48
…OMSPEC case-probe

c496cfb re-derived the win32 expectation from ambient process.env, which
still mismatches production: Node's process.env member access is
case-insensitive on Windows while the child env handed to the planner is a
plain object whose key keeps the OS casing (ComSpec), so the exact
env.COMSPEC probe fell through to the bare-name fallback. The assertion now
derives from the child's OWN captured env/options and probes
COMSPEC/ComSpec/comspec exactly like planClientSpawn does post-dfeea8a11 —
self-contained under any concurrent-test env churn in single-process CI
shards — and additionally pins windowsVerbatimArguments=true on the wrap.
ranxianglei pushed a commit that referenced this pull request Oct 26, 2026
…ed spawn poll (#4646, #4656)

Rebase follow-up on #4646: the branch rewrote the spawn-poll loop inside the
background _bootstrap and thereby dropped the #4656 fix that #4658 implements
on master's structure. Port it into the new loop:

- _find_spawned_in_registry(child_pid) (latest started_at among same-pid
  markers, #4658 review hardening) after the shared-pointer token check and
  before the child-exit check, with the manifest health probe before adoption
- adoption now also records instance_id (the #4658 review nit — /acp panels
  show the adopted instance instead of a blank id)
- tests ported under the new structure: spawn-clobber waits on the bootstrap
  readiness event (register() returns early now) and pins the instance_id;
  registry-find keeps rf_own/rf_foreign/rf_dead/rf_none plus the dup-pid
  aaa-stale vs zzz-own started_at tie-break
@ranxianglei

Copy link
Copy Markdown
Owner Author

closing as superseded by #4646, which merged this branch's registry-pid adoption (latest-started_at hardening + instance_id recording + the rf_* negative tests) inside the restructured background bootstrap. this PR fixed #4656 on the master structure; #4646 carries the same fix plus #4644 self-heal.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working hermes 改动范围:内核之外的主机侧接线(src/agent 适配器、server 路由、注入拦截)/范围兜底/客户端标签 p1 scope/wiring

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[P1][Hermes] Healthy spawned proxy is missed after proxy-origin is overwritten, disabling bili for the host process

2 participants