Repository navigation
[PR] fix(hermes-plugin): restore routed process env on unload and watchdog exit (#4644 follow-up) - #5991
Conversation
(ai: vllm-qwen/qwen3.8-27b) Duplicate screening: not a duplicate. Related to #4644 (the parent: origin-death self-heal that introduced Verified against current master so far:
Initial assessment (full review follows): the |
…y env (ranxianglei#5991) ready-unload scenario: attach to a live origin with a pre-existing HTTPS_PROXY, register until routed, then run the host's unload callback. The process-global env must come back to the host's own values (restored verbatim) and the keys we added (HERMES_CA_BUNDLE/SSL_CERT_FILE) must be dropped — previously _shutdown left the routing at the plugin's origin port, breaking every LLM call in a multiplex host until a full restart.
…y env (ranxianglei#5991) ready-unload scenario: attach to a live origin with a pre-existing HTTPS_PROXY, register until routed, then run the host's unload callback. The process-global env must come back to the host's own values (restored verbatim) and the keys we added (HERMES_CA_BUNDLE/SSL_CERT_FILE) must be dropped — previously _shutdown left the routing at the plugin's origin port, breaking every LLM call in a multiplex host until a full restart.
e0dfcbf to
d7df490
Compare
(ai: vllm-qwen/qwen3.8-27b) Full review done: verified every claim against master, reproduced both the leak and a hazard in the patch itself, and pushed an aligned revision to the branch (head now What holds up
What I changed (aligned revision, 2 commits on your branch)
Behavior change (flagged): Residual observation (not bundled — different mechanism, pre-dates this PR): the stale-thread window itself — Live state: PR head 八问(PR 审核结论)
判定:代码层面审核通过、可合——待新 head CI 转绿后由 owner 批准合并。遗留观察项(stale-thread join 超时窗口)与本 PR 同域但不同机制,未并入;需要的话另行安排。 |
(ai: vllm-qwen/qwen3.8-27b) CI fix run 1/3 — done, pushed. New head What I verified live this run (mirror cache fetched 02:43 UTC): all 12 That is the known inherited master red already reported in #4599 (floors 1151/1152/1155): #5989 ( Fix applied: one labeled commit Local verification: replicated the tripwire scan on the fixed head — 62 live files, zero unregistered spawning suites. Push was fast-forward (remote head re-checked immediately before push, unchanged). CI re-triggers automatically on the push. Note: with the tripwire cleared, these shards will now actually execute tests on this head for the first time — if anything else goes red there (an existing born-red case or a genuine regression in the hermes change), attempt 2/3 will pick it up; nothing else is actionable from this run's evidence. 八问(中文结论)
判定:已推送,等 CI 重跑;shard 转绿后本轮收工。 |
(ai: vllm-qwen/qwen3.8-27b) CI follow-up (verified live just now): the interim red on Live state: head Verdict stands: review passed, no regressions found in the final diff, ready for owner merge whenever you are. |
… exit apply_env() writes HTTPS_PROXY/https_proxy/HERMES_CA_BUNDLE/SSL_CERT_FILE into the process-wide os.environ, but neither _shutdown() (the on_unload callback) nor the watchdog's retire-exit path ever calls _unroute_env(). The docstring assumes 'the host disposes of it' — the host has no knowledge of these keys, so once the routed origin dies and the plugin is disabled/unloaded, the stale HTTPS_PROXY keeps refusing every LLM connection in the process until a full host restart (single-process multiplex hosts take every profile down). Two minimal fixes, both under _lifecycle_lock: - _shutdown(): unroute first when env_applied, then proceed with the existing stop/reap flow. - _watchdog_loop(): the _stopped()/watchdog_gen retire branch unroutes before returning, covering a stop that lands between probe failure and recovery. Verified in isolation (driving the module functions directly): - polluted env -> _shutdown() -> keys removed - pre-existing user proxy in backup -> restored verbatim - env_applied=False (never routed) -> user's own proxy untouched - unpatched build reproduces the leak (env still points at the dead port)
…ds no env work The watchdog-retire unroute added in this PR is redundant in every safe case and harmful in one real race: a stale pre-unload probe (up to HEALTH_PROBE_TIMEOUT_S) can outlive _shutdown's join(timeout=1.0) and retire AFTER a rapid disable+re-enable, unrouting the NEW generation's routing — leaving that generation unrouted with no watchdog, because _ensure_watchdog skips starting one while the stale thread is alive. _shutdown's unroute already covers both leak paths atomically: it runs under the same _lifecycle_lock the verdict block takes, so either _shutdown ran first (env_applied is False when the stale thread retires) or the stop/gen check retires the thread before any env mutation. One unroute site, in _shutdown; comment rewritten upstream-style.
…y env (ranxianglei#5991) ready-unload scenario: attach to a live origin with a pre-existing HTTPS_PROXY, register until routed, then run the host's unload callback. The process-global env must come back to the host's own values (restored verbatim) and the keys we added (HERMES_CA_BUNDLE/SSL_CERT_FILE) must be dropped — previously _shutdown left the routing at the plugin's origin port, breaking every LLM call in a multiplex host until a full restart.
d6256ba to
edf9a74
Compare
(ai: vllm-qwen/qwen3.8-27b) Conflict-resolution run 1/3 — done, pushed. New head What I verified live first:
What I did:
Current state: conflict resolved; CI re-triggers on the new head. The mirror cache still shows the previous head's run, so I can't quote the fresh run's status yet — if anything goes red on 八问(中文结论)
判定:冲突已解、分支已推,等新 head 的 CI 结果;再红则 attempt 2/3 继续。 |
… gate (ranxianglei#5991) The new _shutdown() env gate (if env_applied: _unroute_env()) only had its positive direction pinned by the ready-unload scenario. If the guard were ever removed or made unconditional, _unroute_env() with an empty backup would pop all four routed keys and wipe a host-owned HTTPS_PROXY on unload — undetected, because slow-boot-unload seeded no user proxy (null == null either way). Seed the host's own proxy in that scenario and flip its assertion so both directions of the gate are covered.
(ai: vllm-qwen/qwen3.8-27b) Verdict: APPROVE — correct mechanism, race-safe placement, regression-tested in both directions. Nothing blocks merge; no kernel/ involvement. What I checked
Stale PR body (non-blocking, flagged only) — the "## Fix" section still lists two changes including " Behavior changes (old → new), disclosed:
已顺手修: negative-direction pin for the new 审核八问(中文)
判定:可合(APPROVE)。我的测试提交(2b4e864)推送后 CI 会自动重跑。 |
📦 Merged Build PublishedThis merge to master was published as npm install -g billion-context@0.0.1170-master.58a8796 # this exact build
npm install -g billion-context@master # always the newest merged build
|
Problem
apply_env()routes the host's model traffic by writingHTTPS_PROXY/https_proxy/HERMES_CA_BUNDLE/SSL_CERT_FILEinto the process-wideos.environ._unroute_env()exists (added for #4644) but has exactly one caller: the watchdog's origin-death recovery branch. Two exit paths leave the routed env behind:_shutdown()(theon_unloadcallback) never unroutes. Its docstring says "a registration/env commit either finished before this call (the host disposes of it)" — but the host has no knowledge that the plugin mutatedos.environ, so nothing disposes of it.if _stopped() or watchdog_gen != gen: returnfires before the recovery code, so a plugin disabled after its origin died retires the watchdog silently.Blast radius (real incident)
On a multiplex host where one process serves many profiles, the leaked
HTTPS_PROXYpointing at the dead origin port makes every LLM call in the process fail withConnection refused— all providers, all profiles — until a full process restart. Disabling the plugin as the stop-loss does not help: unload leaves the env in place. We hit exactly this on 2026-11-09 (npm 0.1.212, Hermes gateway, dev-ops profile): xai/zai/deepseek/kimi all refused for ~80 minutes until the gateway was restarted.Fix
Two minimal changes, both under
_lifecycle_lock, both no-ops whenenv_appliedis false:_shutdown(): unroute before the existing stop/reap flow (restores pre-existing values fromenv_backup, or drops the keys)._watchdog_loop(): the retire branch unroutes before returning, covering a stop that lands between a failed probe and recovery.Verification
Driven the module functions directly in isolation (no host):
_shutdown()env_backup→_shutdown()env_applied=False) →_shutdown()The npm 0.1.212
hermes-plugin/__init__.pyis byte-identical to currentmaster, so this patch applies cleanly.