Skip to content

[PR] fix(hermes-plugin): restore routed process env on unload and watchdog exit (#4644 follow-up) - #5991

Merged
ranxianglei merged 4 commits into
ranxianglei:masterfrom
glfruit:fix/hermes-plugin-unroute-env-on-unload
Nov 10, 2026
Merged

ranxianglei merged 4 commits into
ranxianglei:masterfrom
glfruit:fix/hermes-plugin-unroute-env-on-unload

Conversation

@glfruit

@glfruit glfruit commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

Problem

apply_env() routes the host's model traffic by writing HTTPS_PROXY / https_proxy / HERMES_CA_BUNDLE / SSL_CERT_FILE into the process-wide os.environ. _unroute_env() exists (added for #4644) but has exactly one caller: the watchdog's origin-death recovery branch. Two exit paths leave the routed env behind:

  1. _shutdown() (the on_unload callback) never unroutes. Its docstring says "a registration/env commit either finished before this call (the host disposes of it)" — but the host has no knowledge that the plugin mutated os.environ, so nothing disposes of it.
  2. The watchdog's retire branch returns without unrouting. if _stopped() or watchdog_gen != gen: return fires before the recovery code, so a plugin disabled after its origin died retires the watchdog silently.

Blast radius (real incident)

On a multiplex host where one process serves many profiles, the leaked HTTPS_PROXY pointing at the dead origin port makes every LLM call in the process fail with Connection refused — all providers, all profiles — until a full process restart. Disabling the plugin as the stop-loss does not help: unload leaves the env in place. We hit exactly this on 2026-11-09 (npm 0.1.212, Hermes gateway, dev-ops profile): xai/zai/deepseek/kimi all refused for ~80 minutes until the gateway was restarted.

Fix

Two minimal changes, both under _lifecycle_lock, both no-ops when env_applied is false:

  • _shutdown(): unroute before the existing stop/reap flow (restores pre-existing values from env_backup, or drops the keys).
  • _watchdog_loop(): the retire branch unroutes before returning, covering a stop that lands between a failed probe and recovery.

Verification

Driven the module functions directly in isolation (no host):

Scenario Result
env routed to a dead origin → _shutdown() keys removed ✅
user's own proxy recorded in env_backup → _shutdown() restored verbatim ✅
never routed (env_applied=False) → _shutdown() user's own proxy untouched ✅
unpatched build, same scenario env still points at the dead port (leak reproduced) ✅

The npm 0.1.212 hermes-plugin/__init__.py is byte-identical to current master, so this patch applies cleanly.

@ranxianglei ranxianglei changed the title fix(hermes-plugin): restore routed process env on unload and watchdog exit (#4644 follow-up) [PR] fix(hermes-plugin): restore routed process env on unload and watchdog exit (#4644 follow-up) Nov 9, 2026
@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Duplicate screening: not a duplicate. Related to #4644 (the parent: origin-death self-heal that introduced _unroute_env) but a distinct failure domain — #4644 is "routed origin dies → no recovery"; this is "plugin unloads / watchdog retires → routed env leaks". No other open issue or PR covers unload-time env restoration; #5116/#5117 (plain-http base_url routing) touch env routing but via a different mechanism (attach path, not lifecycle teardown).

Verified against current master so far:

  • Your byte-identity claim holds: hermes-plugin/ is unchanged on master since v0.1.212 (git log v0.1.212..master -- hermes-plugin/ is empty), so the patch applies cleanly.
  • _unroute_env() does have exactly one caller today — the watchdog's origin-death recovery branch (hermes-plugin/__init__.py:1309). Both leak paths you describe are real in the code: _shutdown() never unroutes, and the retire branch returns first.
  • CI on the PR head is running; no failures reported so far (13 of 30 checks still pending at time of writing).

Initial assessment (full review follows): the _shutdown() hunk is correct and, by itself, closes both leak paths you list — _shutdown()'s unroute+stop+gen-bump happens atomically under the same RLock the watchdog takes, so once it lands there is no interleaving where the retire branch fires with routed env still applied. That makes the watchdog-retire hunk redundant in every safe case — and I believe it is actively harmful in one narrow race (rapid disable → re-enable while a stale pre-unload watchdog thread is still finishing a probe): the stale thread would retire after the new generation re-applied its env, and the added unroute would then destroy the new routing, leaving the process unrouted with no live watchdog to recover it (the new gen never got one because _ensure_watchdog skips starting a thread while the stale one is still alive). I'm reproducing this reasoning against the code locally right now; if it holds I will push an aligned revision to your branch (keep the _shutdown fix, drop the retire hunk, rewrite the "LOCAL PATCH" comments into upstream-style ones) plus a regression test through the existing tests/hermes-native.test.ts subprocess harness — manual verification alone doesn't satisfy the repo's regression-test bar. Full structured review comment coming next.

@ranxianglei ranxianglei added bug Something isn't working p1 hermes 改动范围:内核之外的主机侧接线(src/agent 适配器、server 路由、注入拦截)/范围兜底/客户端标签 scope/wiring labels Nov 9, 2026
ranxianglei pushed a commit to glfruit/billion-context that referenced this pull request Nov 9, 2026
…y env (ranxianglei#5991)

ready-unload scenario: attach to a live origin with a pre-existing HTTPS_PROXY,
register until routed, then run the host's unload callback. The process-global env
must come back to the host's own values (restored verbatim) and the keys we added
(HERMES_CA_BUNDLE/SSL_CERT_FILE) must be dropped — previously _shutdown left the
routing at the plugin's origin port, breaking every LLM call in a multiplex host
until a full restart.
ranxianglei pushed a commit to glfruit/billion-context that referenced this pull request Nov 9, 2026
…y env (ranxianglei#5991)

ready-unload scenario: attach to a live origin with a pre-existing HTTPS_PROXY,
register until routed, then run the host's unload callback. The process-global env
must come back to the host's own values (restored verbatim) and the keys we added
(HERMES_CA_BUNDLE/SSL_CERT_FILE) must be dropped — previously _shutdown left the
routing at the plugin's origin port, breaking every LLM call in a multiplex host
until a full restart.
@ranxianglei
ranxianglei force-pushed the fix/hermes-plugin-unroute-env-on-unload branch from e0dfcbf to d7df490 Compare November 9, 2026 02:35
@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Full review done: verified every claim against master, reproduced both the leak and a hazard in the patch itself, and pushed an aligned revision to the branch (head now d7df4903, rebased onto current master 1f311dd8).

What holds up

  • The leak is real and the incident analysis is accurate: apply_env() writes HTTPS_PROXY/https_proxy/HERMES_CA_BUNDLE/SSL_CERT_FILE process-wide (hermes-plugin/__init__.py:1202), and _unroute_env() had exactly ONE caller (the watchdog's origin-death recovery, :1309) — so both exit paths leaked. Reproduced on master: after _shutdown() the env still points at the dead origin port.
  • The npm 0.1.212 byte-identity claim checks out: hermes-plugin/ is unchanged since that tag.
  • The _shutdown() unroute (hunk 1) is correct and sufficient for both leak paths: it runs under the same RLock the watchdog verdict block takes, so shutdown-vs-watchdog ordering is atomic — either env_applied is already False when the watchdog retires, or the stop/gen check retires the thread before any env mutation.

What I changed (aligned revision, 2 commits on your branch)

  1. Dropped the watchdog-retire unroute (hunk 2). It is redundant in every safe case and harmful in one real race: a stale pre-unload probe outlives _shutdown's join(timeout=1.0) — a probe blocks up to HEALTH_PROBE_TIMEOUT_S=2.0, and a recovery re-bootstrap runs up to 90s. After a rapid disable→re-enable, the stale thread finishes mid-probe, sees the gen mismatch, and unroutes the NEW generation's routing — leaving it unrouted with no watchdog, because _ensure_watchdog skips starting one while the stale thread is still alive. Empirically confirmed: that scenario fails with the original two-hunk patch and passes with the single-site fix.
  2. Rewrote the hunk-1 comment upstream-style (the "LOCAL PATCH …" labels don't belong in-tree).
  3. Added the regression test the PR was missing: new ready-unload scenario in tests/hermes-native.test.ts — attach with a pre-existing user proxy, register until routed, unload, assert the user proxy is restored verbatim and the CA keys we added are dropped. Fails on master (reproduces the leak), passes on the fixed head; full file suite 52/52 green locally.

Behavior change (flagged): _shutdown() now touches os.environ — before: unload left all four routed keys in place (the bug); after: pre-existing values restored verbatim, keys the plugin added are dropped; no-op when env was never applied. That IS the fix.

Residual observation (not bundled — different mechanism, pre-dates this PR): the stale-thread window itself — join(timeout=1.0) < probe timeout (2.0s) plus the alive-check skip in _ensure_watchdog — can strand a generation without its origin-death watcher even without any env mutation. Worth a separate look if you want it tracked.

Live state: PR head d7df4903, rebased clean (no overlap between the 7 new master commits and these files); CI for the new head is running — the previous head f3ca0c7a finished 30/30 green. I'll confirm when the new run lands.


八问(PR 审核结论)

  1. 是什么问题:Hermes 插件卸载时把进程级代理环境变量(HTTPS_PROXY 等四项)留在已死的 origin 端口上;多 profile 宿主里该进程内所有 LLM 调用全部 Connection refused,直到重启进程——禁用插件止血无效。
  2. 什么原因导致的:apply_env() 写的是进程全局 os.environ,而两条退出路径(_shutdown 卸载回调、看门狗退役分支)都不调用 _unroute_env()——全仓只有 origin-death 恢复一个调用点;宿主不知道插件动过环境变量,不会替它清理。
  3. 影响范围与严重程度:p1。触发条件 = 插件曾成功路由流量 + 随后被卸载/禁用;后果 = 整进程全部 provider 的 LLM 调用失败(2026-11-09 真实事故约 80 分钟),且止血手段(禁用插件)本身失效。
  4. 怎么修复的:在 _shutdown() 锁内、stop/reap 流程之前调用 _unroute_env()(还原备份值或删除新增键)。单一 unroute 点在 RLock 下与看门狗裁决块互斥,原子覆盖两条泄漏路径;看门狗退役分支不加 unroute——它在该分支可达时必然已无 env 可清。
  5. 是否有回归问题:逐项审过——never-routed 时 _shutdown 为 no-op(用户自有代理不受影响)✅;origin-death 恢复路径行为不变 ✅;slow-boot-unload / re-enable / attach 等既有场景测试全绿 ✅。原 PR 第二处改动(退役分支 unroute)在「快速禁用→重新启用」竞态下会摧毁新一代路由且无看门狗兜底(实测复现),属真回归风险,已移除。
  6. 是否破坏缓存:不涉及——只改卸载时刻的环境变量清理,不碰缓存/重试/游标/去重键。
  7. 是否改了设计方向:没有。env 路由、看门狗、bootstrap 机制全部不动,只在既有生命周期终点补上对称的清理动作。
  8. 是否引入了新的配置:无新配置、无新 env、无新默认值。

判定:代码层面审核通过、可合——待新 head CI 转绿后由 owner 批准合并。遗留观察项(stale-thread join 超时窗口)与本 PR 同域但不同机制,未并入;需要的话另行安排。

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

CI fix run 1/3 — done, pushed. New head d6256babe1 (was d7df490311).

What I verified live this run (mirror cache fetched 02:43 UTC): all 12 test (…) shard jobs failed on head d7df490311; every other check green or skipped-by-design. I pulled a failing job's log (job 128949052362, ubuntu-24 shard 3) and checked it line by line: the shard exits at pretest with exactly one error and zero test output —

process-spawning test files missing from LIVE_FILES in scripts/run-ci-tests.mjs: tests/issue5955-wedged-same-lane-takeover.test.ts. Register them there after reviewing their concurrency assumptions (#1248).

That is the known inherited master red already reported in #4599 (floors 1151/1152/1155): #5989 (016d933a0, merged into master as 48f9e6916) landed that spawning suite without registering it in scripts/live-files.mjs. Master pushes skip the heavy test lanes, so master looks green while every PR head containing that commit dies at the tripwire. It is not caused by this PR's diff — the hermes-plugin env-restore change touches neither live-files.mjs nor that suite.

Fix applied: one labeled commit d6256babe1 appending the registration to the append-only LIVE_FILES zone — byte-identical to in-flight PR #5990's hunk, so once #5990 merges it dedupes cleanly on the next rebase (no duplicate line ever lands on master). Per the few-line CI-fix discipline (better a duplicated in-branch fix than waiting out the merge queue); same practice as floors 1151/1155. I deliberately did NOT also copy #5990's second hunk (the observations-assertion flake tolerance in the suite itself) — that semantic change belongs to #5990's scope; if that assertion flakes once shards actually run, it will surface as a real test failure and get handled then.

Local verification: replicated the tripwire scan on the fixed head — 62 live files, zero unregistered spawning suites. Push was fast-forward (remote head re-checked immediately before push, unchanged). CI re-triggers automatically on the push. Note: with the tripwire cleared, these shards will now actually execute tests on this head for the first time — if anything else goes red there (an existing born-red case or a genuine regression in the hermes change), attempt 2/3 will pick it up; nothing else is actionable from this run's evidence.


八问(中文结论)

  1. 是什么问题:本 PR 的 12 个 test shard 全部在跑任何测试之前就退出——CI tripwire(tests/claude-native.test.ts linux live-E2E subtests fail/hang in shared multi-agent sandboxes #1248,SPAWN_RE/LIVE_FILES)判定 tests/issue5955-wedged-same-lane-takeover.test.ts 会起子进程但未登记,pretest 自检直接中止每个分片。
  2. 什么原因导致的:[PR] test(launcher): cover the #5955 wedged-takeover escalation ladder + win32 ESRCH silence #5989(016d933a0,以 48f9e6916 合入 master)把该测试文件带进仓库却没登记进 scripts/live-files.mjs;master push 不跑重型 test 车道,所以 master 面绿、所有含该 commit 的 PR head 继承这条红(job 日志逐行核对,与 CI Control Center: Centralized CI Failure Handling & Status Sync (Comment all CI issues here; do not open new issues) | CI 总控: CI 失败集中处理与状态同步(所有 CI 问题在此评论, 勿开新 issue) #4599 floor 1151/1152/1155 同签名)。与本 PR 自身 diff 无关。
  3. 影响范围与严重程度:所有在途 PR 的测试矩阵(信号级停摆,非用户侧功能问题);本 PR 因此一直合不了。
  4. 怎么修复的:按几行级小修纪律,在本分支加独立标注 commit d6256babe1,把该文件登记进 LIVE_FILES;与在途 [PR] test(launcher): tolerate the documented one-observation loss in the #5955 takeover ledger #5990 的登记逐字节一致,[PR] test(launcher): tolerate the documented one-observation loss in the #5955 takeover ledger #5990 合入后下次 rebase 自动去重,master 不会留下重复行。
  5. 是否有回归问题:无。纯 append-only 追加(既有条目零改动、零重排),本地 tripwire 复扫通过(62 项全存在、零漏登),未触碰其他文件。
  6. 是否破坏缓存:不涉及。
  7. 是否改了设计方向:没有;根治收敛在 [PR] test(launcher): tolerate the documented one-observation loss in the #5955 takeover ledger #5990/ci-fixes,本分支只解锁自己。
  8. 是否引入了新的配置:没有。

判定:已推送,等 CI 重跑;shard 转绿后本轮收工。

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

CI follow-up (verified live just now): the interim red on d7df4903 was an inherited master red, not this PR — tests/issue5955-wedged-same-lane-takeover.test.ts (landed via #5989) was never registered in scripts/live-files.mjs, so the run-ci-tests.mjs tripwire killed all 11 test shards before any test ran; it fires identically on bare master 1f311dd8. The sentinel had already converged the one-line fix on ci-fixes (831b6458), and @glfruit beat me to it by pushing the byte-identical registration as commit d6256babe1 on the PR branch — I verified equivalence and did not duplicate the fix.

Live state: head d6256babe1, all 10 workflows green including the main CI test lanes (confirmed via the run list; the mirror cache is lagging behind). Final branch shape: author's original fix → aligned revision (single unroute site in _shutdown, upstream-style comment) → regression test ready-unload → CI registration fix.

Verdict stands: review passed, no regressions found in the final diff, ready for owner merge whenever you are.

glfruit and others added 3 commits November 10, 2026 14:14
… exit

apply_env() writes HTTPS_PROXY/https_proxy/HERMES_CA_BUNDLE/SSL_CERT_FILE into
the process-wide os.environ, but neither _shutdown() (the on_unload callback)
nor the watchdog's retire-exit path ever calls _unroute_env(). The docstring
assumes 'the host disposes of it' — the host has no knowledge of these keys,
so once the routed origin dies and the plugin is disabled/unloaded, the stale
HTTPS_PROXY keeps refusing every LLM connection in the process until a full
host restart (single-process multiplex hosts take every profile down).

Two minimal fixes, both under _lifecycle_lock:
- _shutdown(): unroute first when env_applied, then proceed with the existing
  stop/reap flow.
- _watchdog_loop(): the _stopped()/watchdog_gen retire branch unroutes before
  returning, covering a stop that lands between probe failure and recovery.

Verified in isolation (driving the module functions directly):
- polluted env -> _shutdown() -> keys removed
- pre-existing user proxy in backup -> restored verbatim
- env_applied=False (never routed) -> user's own proxy untouched
- unpatched build reproduces the leak (env still points at the dead port)
…ds no env work

The watchdog-retire unroute added in this PR is redundant in every safe case and
harmful in one real race: a stale pre-unload probe (up to HEALTH_PROBE_TIMEOUT_S) can
outlive _shutdown's join(timeout=1.0) and retire AFTER a rapid disable+re-enable,
unrouting the NEW generation's routing — leaving that generation unrouted with no
watchdog, because _ensure_watchdog skips starting one while the stale thread is alive.

_shutdown's unroute already covers both leak paths atomically: it runs under the same
_lifecycle_lock the verdict block takes, so either _shutdown ran first (env_applied is
False when the stale thread retires) or the stop/gen check retires the thread before any
env mutation. One unroute site, in _shutdown; comment rewritten upstream-style.
…y env (ranxianglei#5991)

ready-unload scenario: attach to a live origin with a pre-existing HTTPS_PROXY,
register until routed, then run the host's unload callback. The process-global env
must come back to the host's own values (restored verbatim) and the keys we added
(HERMES_CA_BUNDLE/SSL_CERT_FILE) must be dropped — previously _shutdown left the
routing at the plugin's origin port, breaking every LLM call in a multiplex host
until a full restart.
@ranxianglei
ranxianglei force-pushed the fix/hermes-plugin-unroute-env-on-unload branch from d6256ba to edf9a74 Compare November 10, 2026 06:26
@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Conflict-resolution run 1/3 — done, pushed. New head edf9a74fb3, rebased onto current master 4dea18eabf. Conflict cleared: live API check shows mergeable: true (state was false/dirty before).

What I verified live first:

What I did:

  1. Rebased the branch onto origin/master. Zero conflicts — git auto-merged everything. Notably, my LIVE_FILES registration commit (d6256babe1) was automatically dropped (skipped previously applied) because [PR] test(launcher): tolerate the documented one-observation loss in the #5955 takeover ledger #5990 merged into master with a byte-identical hunk — the planned dedupe happened exactly as designed, so master carries the line exactly once.
  2. Post-rebase diff vs master is back to the PR's original footprint only: hermes-plugin/__init__.py (+7) and tests/hermes-native.test.ts (+31), purely additive.
  3. Post-rebase verification per §7.2: npm run typecheck OK, npm run build OK, targeted suite tests/hermes-native.test.ts 53/53 pass (includes the new [PR] fix(hermes-plugin): restore routed process env on unload and watchdog exit (#4644 follow-up) #5991 unload-env regression test).
  4. Pushed with --force-with-lease (lease = remote-tracking ref d6256babe1, verified matching the live remote head seconds before the push — the one case where force-push is authorized here is exactly this conflict-rebase).

Current state: conflict resolved; CI re-triggers on the new head. The mirror cache still shows the previous head's run, so I can't quote the fresh run's status yet — if anything goes red on edf9a74fb3 (e.g. something master-side changed under the rebased tree), attempt 2/3 picks it up. No action needed from anyone unless the owner wants to look at the (now clean) diff before merging.


八问(中文结论)

  1. 是什么问题:PR 分支停在旧 master(1f311dd8d5),master 已推进到 4dea18eabf(评审期间合入约 30 个 PR),PR 与 base 冲突(dirty),合并按钮不可用。
  2. 什么原因导致的:不是代码分歧——是时间差:PR 在途期间 master 持续合入([PR] test(launcher): tolerate the documented one-observation loss in the #5955 takeover ledger #5990、[PR] ci: master-CI convergence branch (sentinel-maintained, per #4599) #6073 ci-fixes 等),分支没有跟上;base 前进后 GitHub 判 dirty。上一轮修的 CI 红(tripwire)本身已解决:d6256babe1 上 11 个 test shard 全绿。
  3. 影响范围与严重程度:仅本 PR 无法合并(流程层面),无用户侧影响。
  4. 怎么修复的:rebase 到当前 master,零人工冲突。上轮加的 LIVE_FILES 登记 commit 因与已合入的 [PR] test(launcher): tolerate the documented one-observation loss in the #5955 takeover ledger #5990 逐字节相同被 git 自动丢弃("skipped previously applied")——按计划自然去重,master 上该行只有一份。
  5. 是否有回归问题:无。rebase 后 diff 回到本 PR 原始足迹(两个文件、纯新增 38 行);本地 typecheck + build + 定向套件 53/53 绿。
  6. 是否破坏缓存:不涉及。
  7. 是否改了设计方向:没有。
  8. 是否引入了新的配置:没有。

判定:冲突已解、分支已推,等新 head 的 CI 结果;再红则 attempt 2/3 继续。

… gate (ranxianglei#5991)

The new _shutdown() env gate (if env_applied: _unroute_env()) only had its positive direction pinned by the ready-unload scenario. If the guard were ever removed or made unconditional, _unroute_env() with an empty backup would pop all four routed keys and wipe a host-owned HTTPS_PROXY on unload — undetected, because slow-boot-unload seeded no user proxy (null == null either way). Seed the host's own proxy in that scenario and flip its assertion so both directions of the gate are covered.
@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Verdict: APPROVE — correct mechanism, race-safe placement, regression-tested in both directions. Nothing blocks merge; no kernel/ involvement.

What I checked

  • Full diff (+38 lines, 2 files) plus the surrounding lifecycle code in hermes-plugin/__init__.py at PR head (apply_env / _unroute_env / _watchdog_loop / _shutdown / _reap_unready_child).
  • Proved the regression test locally in both directions: with the unpatched master plugin the new ready-unload scenario fails exactly like the incident (HTTPS_PROXY still pointing at the dead sidecar port after unload); with the PR plugin it passes — the host proxy is restored byte-for-byte and the bili-added CA keys are dropped.
  • Race audit: the new guard runs inside the same _lifecycle_lock as apply_env() and the watchdog recovery path, so no interleaving can leave env applied past stop; a double-unload is idempotent (guard reads False on the second pass); backup semantics verified (restore-if-backed-up, else pop).
  • Gate coverage (gate-test discipline Test Blind Spot Retrospective: Reinforcing Negative Assertion Discipline, Tightening Fake Upstream Strictness, Thin Inst | 测试盲区复盘:补反向断言纪律、假上游严格化、薄安装冒烟、形态矩阵与生命周期覆盖 #3617): the positive direction was pinned by the author's test; the negative direction (an unload that never routed must not touch host env) was missing — fixed below.

Stale PR body (non-blocking, flagged only) — the "## Fix" section still lists two changes including "_watchdog_loop(): the retire branch unroutes before returning", but commit c5f742a removed that change from the diff. I independently verified the removal is correct: every retire path returns either because _stopped() (⇒ _shutdown() already ran, i.e. env already unrooted under the same lock hold) or because the generation changed (⇒ the new generation owns env), and apply_env() refuses once stopped — so the retire branch genuinely needs no env work. Body text left untouched (author's words); the final code matches c5f742a's rationale.

Behavior changes (old → new), disclosed:

  1. Unload with routed env — the four keys HTTPS_PROXY/https_proxy/HERMES_CA_BUNDLE/SSL_CERT_FILE previously stayed pointed at the dead sidecar origin until the host process restarted (the reported incident: ~80 min of all-providers Connection refused in a multiplexed Hermes gateway) → now restored to host values, or dropped where bili added them. This is the fix itself.
  2. Spawn-mode child on unload — previously survived an unload while env was applied (_reap_unready_child() early-returns on env_applied) → now terminated at the end of _shutdown(), since routing is removed first and nothing points at the child anymore. Intended: unload leaves nothing behind; attach mode (child=None) is unaffected.

已顺手修: negative-direction pin for the new _shutdown env gate — seeded the host's own proxy into the existing slow-boot-unload scenario and flipped its env_after_unload assertion from null to the user value, so a future unconditional _unroute_env() (which would pop all four keys off an empty backup and wipe a real host proxy) is caught by CI. Green on the PR head and on master (forward-looking pin only). + commit 2b4e864


审核八问(中文)

  1. 是什么问题:hermes native 插件卸载时不回滚进程级代理环境变量——HTTPS_PROXY/https_proxy/HERMES_CA_BUNDLE/SSL_CERT_FILE 继续指向已死的 sidecar 端口,直到宿主进程重启;多路复用网关场景下一次事故全 provider 连不上约 80 分钟(bug(hermes): No self-healing after bootstrap failure/agent death; silent direct connection can persist for the entire ho | bug(hermes): 引导失败/代理死亡后无自愈,静默直连可持续整个宿主生命周期 #4644 后续)。
  2. 什么原因导致的:apply_env() 把路由写进 os.environ 并备份原值,但卸载路径 _shutdown() 从不调用 _unroute_env()——env 的写入方与清理方不对称,生命周期只有「装」没有「卸」。
  3. 影响范围与严重程度:所有用 hermes native 插件且宿主进程长期存活(多路复用网关、常驻 agent)的用户面;unload 之后到进程重启之前,走这些环境变量的出站流量全部指向死端口 = 该窗口内完全不可用(p1 级症状);attach 模式不受影响。
  4. 怎么修复的:在 _shutdown() 里同一把 _lifecycle_lock 下、stop_event 置位之前加 if env_applied: _unroute_env()——复用既有备份/还原机制(有备份则还原、无备份则 pop),与 apply_env()/watchdog 恢复共用同一把锁串行化,无双载竞态。
  5. 是否有回归问题:审过的既有行为——①watchdog 退休分支不需要补 unroute(所有退休路径要么 _stopped() 即 shutdown 已清、要么代际不匹配即新代际持有 env),c5f742a221 删掉 v1 的 watchdog 改动是对的;②spawn 模式子进程现在会在 unload 时被回收(旧行为是残留存活,因为 env_applied=True 时 _reap_unready_child() 提前返回)——有意变更,见上披露;③从未路由过的会话 unload 不动任何键(负向方向已由我补的测试钉住)。无意外回归。
  6. 是否破坏缓存:不涉及——纯进程环境变量生命周期修复,不触碰会话存储/压缩缓存/重试游标。
  7. 是否改了设计方向:不涉及——沿用既有 apply/unroute 对称机制与单锁纪律,只补上缺失的卸载半边。
  8. 是否引入了新的配置:不涉及。

判定:可合(APPROVE)。我的测试提交(2b4e864)推送后 CI 会自动重跑。

@ranxianglei
ranxianglei merged commit 58a8796 into ranxianglei:master Nov 10, 2026
48 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

📦 Merged Build Published

This merge to master was published as 0.0.1170-master.58a8796 (rolling master tag).

npm install -g billion-context@0.0.1170-master.58a8796  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working hermes 改动范围:内核之外的主机侧接线(src/agent 适配器、server 路由、注入拦截)/范围兜底/客户端标签 p1 scope/wiring

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants