Repository navigation
[PR] fix(hermes): adopt own spawned child via registry when the shared proxy-origin pointer is clobbered - #4658
ranxianglei wants to merge 15 commits into
Conversation
…s clobbered (#4656) The Hermes plugin's ensure_origin spawn-wait loop only trusted the single machine-wide proxy-origin pointer and required its launchToken to equal this round's token. A concurrent bring-up registering AFTER our child overwrites that last-writer-wins pointer with a foreign token, so the match starves for the whole startup budget and the loop SIGKILLs our healthy child at timeout -> the host goes inert (model traffic direct, uncompressed). Mirror the TypeScript launcher fix (#2954): after the shared-pointer check, scan the per-instance registry markers for OUR child by pid (the registry omits the launcher-private token and is keyed per instance, so it cannot be clobbered) and adopt its origin once a health probe passes. It runs before the child-exit check so a healthy self-registration wins even if the child process has already exited by this poll. Regression coverage: the spawn-clobber scenario reproduces the clobber end-to-end and fails pre-fix; the _find_spawned_in_registry unit test pins the own-pid-only acceptance (a live foreign instance with its own launchToken and a stale dead-pid marker are both rejected).
📦 Built Package ArtifactBranch: Latest build — install this exact versionnpm install -g billion-context@0.0.5973-pr.4658.8f5f618This exact version is immutable and reproducible. The rolling Recent builds (commit → version)
Download artifact
tar xzf billion-context-pr4658.tgz
npm install -g packageInstall instructions are refreshed on each push. |
… share the spawned pid (#4658 review) Review follow-up on the #4656 adoption helper: _find_spawned_in_registry returned the first pid-matching marker in filename order. If a stale marker from a previous run whose dead pid the OS later reused for our child sorts before our own marker (and its recorded origin happens to still answer healthy probes), the first-match rule would adopt the foreign origin. Among same-pid matches the newest started_at now wins — our child's own marker always postdates any stale one. The TS twin (#2954, src/launcher.ts) closes the same corner with a responder-pid check from /__bili/health; this keeps the two lanes aligned in spirit without adding an extra HTTP probe per poll. Test pins the tie-break with a dup-pid sub-case where the stale marker sorts alphabetically first.
(ai: vllm/qwen3.8-27b) 查重: 同症状同机制的竞争 PR 存在——#4657(外部贡献)。两者都修「共享 proxy-origin 指针被并发启动覆写后认不到自己刚拉起的健康子进程」。机制比对:#4657 要求每个候选(共享指针 + 全部 registry 条目) 标签: bug, hermes, scope/wiring, p1(并发拉起时 Hermes 宿主整体 inert 的 P1 可用性缺陷;hermes 客户端;协议接线层)——保留既有 pr 标签。 结构化八问
行为变更(PR 已声明,复核确认):旧=clobber 场景下耗尽预算→杀健康子进程→inert 直到手动重启;新=经 registry 采纳自己的健康子进程,压缩保持活跃。属恢复设计意图,其他代码路径不受影响。 审核发现并已直接修(小问题直接修): 验证证据:
判定:可合(待 test 分片全绿;本地定向验证 + 前置失败复现均已完成,竞争 PR #4657 建议关闭)。 |
(ai: vllm/qwen3.8-27b) CI 跟进:新 head |
…ment #4316 PR3/3 (restampRefsToLedger) re-homes citation numbers once on every full-mode cold load; goldens pinning exact ref numbers are born-red. Re-pin the 7 storage-lane tests (engine-proofs x6 + orphan-recovery x1) modulo that renumbering — zero production code touched. - golden-storage-drive.ts: per-turn citation-normalized digest (shaCiteNorm) and state digest modulo stripOneTimeAlignment (stateDigestAligned); the deep-sort helper is hoisted into golden-canonical.ts for sharing. - corrupt-index-child.ts: new stripOneTimeAlignment normalizer — strips the refLedgerAligned stamp and re-homes refs onto canonical raw-id order so a clean full-mode read and the legacy fallback load compare equal. - storage-engine-proofs.test.ts: wire legs assert shaCiteNorm equality vs the legacy restart leg and the committed goldens; state legs assert stateDigestAligned; rollback leg now spells BILI_STORAGE_UNIFIED=legacy explicitly (#3818 flip: absent resolves to full, deleting the env would vacate the legacy path). - storage-orphan-recovery.test.ts: pins the content-addressable byRaw->byRef round-trip instead of a counter-era m00003 literal. - golden/storage-render/*.json regenerated with shaCiteNorm/stateDigestAligned. Converges the master-red storage-lane family (#4599 plan A, owner directive).
…dger parity masks The #4316 one-time renumbering restamped messageRefs but left state.tokenSnapshot keyed by the OLD refs, so post-restart bodies carried stale per-message token counts (1.3K vs 1.4K) and the full-mode-restart wire drifted off the legacy baseline (#4599). persist.ts: two-pass remap (stage moves, delete old keys, then write new keys) so entries whose new ref collides with another move's old ref survive; fires only when superseded refs exist. Aligned normalizer (corrupt-index-child.ts): the cacheLedger's segment bookkeeping (be/bl/bs, unfinalized tail line), the derived agg block, and per-line miss-attribution (prov/seam) are restart experience, not content — no-restart and restart legs legitimately differ there. Masked in the aligned view only; raw digests keep full accounting. Goldens regenerated (stateDigest + aligned). storage-engine-proofs 12/12, golden-storage-render 5/5 locally.
…persisted state (#4599) The fork-survival test asserted immediately after the compress tool response, but the fold lands in the live session state asynchronously (#3573 replay semantics). Slow CI runners persisted the pre-fold state ('0 block(s) (compressions=0)' cold load) and the deepEqual diverged. Bounded 5s wait on the in-memory fold; fast machines exit on the first iteration. Product semantics unchanged.
The #4316 cold-load alignment reads the unified sidecar; its ingest of just-flushed records completes asynchronously. Slow runners compared the pre-ingest state and saw refLedgerAligned missing while the fold itself had landed. Poll (bounded 5s) until the loaded state carries the flag; fast machines exit on the first probe.
Git-for-Windows autocrlf checks the bundled .md posts out with CRLF; the exact-match '---' fences then failed to parse and every post lost its title/excerpt/date frontmatter (Windows CI: 'bundled meet-bili post found' assertion). Normalize CRLF to LF inside parseFrontmatter when the fence is CRLF-shaped; LF content is byte-identical as before. Regression test with a CRLF fixture.
… storage mode (#4599) The #4316 cold-load alignment sets refLedgerAligned only when the unified store is available; config-clean machines (CI runners) run legacy mode where it is an honest skip. The test asserted the flag unconditionally, so every shard carrying this file was born-red on CI while unified-config machines passed. Expectation now mirrors store.replicasAvailable(); verified 49/49 in BOTH modes locally.
…mspec wrap on win32 (#4599)
The ci-check lane's expectation change (c496cfb) exposed a real Windows bug: stripInheritedProxy builds inlineEnv as a spread copy of process.env, which keeps the original key casing (ComSpec) — Windows env vars are case-insensitive but plain objects are not. planClientSpawn's exact env.COMSPEC probe then missed it and silently degraded every wrapped spawn to the bare-name 'cmd.exe' fallback (verified by repro on a real win32 box: childEnv.COMSPEC=undefined while process.env.COMSPEC was set). Fix: probe COMSPEC/ComSpec/comspec in planClientSpawn and planDshSpawn. Verified on real Windows (Node 22): launcher-opencode-inline + fix-679 + dsh-refresh 37/37; host tsc clean, 35 pass / 2 win32-skips.
|
补一条真实环境现场,支持按 pid 识别的方向。 本机 2026-10-25(CST):三个代理分别在 46291 / 45507 / 48383 监听成功,约一分钟后插件仍报 一条边界说明,避免误导后来者:#4657 按 |
(ai: vllm/qwen3.8-27b) 感谢补充现场证据。三方并发拉起的场景与 #4656 的触发条件完全吻合:共享 边界说明与审核结论一致:真实 registry marker 不含 当前状态:#4658 CI 全绿(含一条审核跟进 commit |
…OMSPEC case-probe c496cfb re-derived the win32 expectation from ambient process.env, which still mismatches production: Node's process.env member access is case-insensitive on Windows while the child env handed to the planner is a plain object whose key keeps the OS casing (ComSpec), so the exact env.COMSPEC probe fell through to the bare-name fallback. The assertion now derives from the child's OWN captured env/options and probes COMSPEC/ComSpec/comspec exactly like planClientSpawn does post-dfeea8a11 — self-contained under any concurrent-test env churn in single-process CI shards — and additionally pins windowsVerbatimArguments=true on the wrap.
…ed spawn poll (#4646, #4656) Rebase follow-up on #4646: the branch rewrote the spawn-poll loop inside the background _bootstrap and thereby dropped the #4656 fix that #4658 implements on master's structure. Port it into the new loop: - _find_spawned_in_registry(child_pid) (latest started_at among same-pid markers, #4658 review hardening) after the shared-pointer token check and before the child-exit check, with the manifest health probe before adoption - adoption now also records instance_id (the #4658 review nit — /acp panels show the adopted instance instead of a blank id) - tests ported under the new structure: spawn-clobber waits on the bootstrap readiness event (register() returns early now) and pins the instance_id; registry-find keeps rf_own/rf_foreign/rf_dead/rf_none plus the dup-pid aaa-stale vs zzz-own started_at tie-break
|
closing as superseded by #4646, which merged this branch's registry-pid adoption (latest-started_at hardening + instance_id recording + the rf_* negative tests) inside the restructured background bootstrap. this PR fixed #4656 on the master structure; #4646 carries the same fix plus #4644 self-heal. |
What & why
The Hermes plugin (
hermes-plugin/__init__.py) spawns its own bili proxy, then polls for it to become healthy. That poll only trusted the single machine-wideproxy-originpointer and required itslaunchTokento equal this round's token.That pointer is last-writer-wins. A concurrent bring-up (another lane starting, a second starter) that registers after our child overwrites it with a foreign token, so the launchToken match starves for the whole startup budget (
SPAWN_WAIT_S). At timeout the loop callschild.terminate()— killing our healthy child — and the host goes inert: model traffic goes direct (uncompressed). That is exactly the failure reported in #4656. Our healthy instance was correctly registered in its own per-instance marker (instances/<id>.json) all along; the poll simply never looked there.Fix
Mirror the TypeScript launcher fix already shipped for this exact mechanism (#2954). After the shared-pointer check, scan the per-instance registry markers for our child by
pidand, once a health probe of the origin it recorded passes, adopt it:pid, notlaunchToken, because the registry marker deliberately omits the launcher-private token (src/server.tsstrips it); a pid we just spawned cannot be faked by a concurrent writer.Behavior change (intended, flagged)
proxy-originwas clobbered by a concurrent bring-up, the Hermes plugin timed out, terminated its healthy child, and stayed inert (uncompressed direct traffic) until a manual restart.Tests
spawn-clobber(new): end-to-end repro — pre-seeds a foreign clobbered pointer, the spawned child self-registers by pid; asserts the plugin becomes active (tools registered) withmode="spawn". Fails pre-fix (times out / stays inert), passes post-fix — verified by temporarily disabling the fix line._find_spawned_in_registry(new): pins the own-pid-only acceptance — a live foreign instance carrying its ownlaunchTokenand a stale dead-pid marker are both rejected; no child pid → never match. Deterministic (no subprocess).Full
tests/hermes-native.test.tsgreen on the rebased head (40/40); typecheck clean.Closes #4656