Skip to content

[PR] fix: drop stale host proxy hops from intercepted dispatches (#4566 part 1) - #4878

Merged
ranxianglei merged 6 commits into
masterfrom
2026-10-25_fix-4566-stale-proxy-hop
Nov 9, 2026
Merged

ranxianglei merged 6 commits into
masterfrom
2026-10-25_fix-4566-stale-proxy-hop

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

Part 1 of #4566 — the host's cached proxy hop surviving lane re-targeting. Part 2 (the lane itself moving under serving hosts) awaits the decision requested in the issue thread (A1 vs A2) and will land as its own PR.

What was wrong

A plugin host memoizes its proxy hop for its whole process life (Bun init.proxy stamped from PI_PROXY_ANTHROPIC, plus the launcher-exported HTTP(S)_PROXY). When a lane port moves under live hosts (ladder drift), that stale hop rides every dispatch the intercept performs:

  • rewritten/self/rerouted /bili/ targets dialed through the dead port → connection refused on every model turn;
  • direct fallbacks kept the hop even when it pointed at a bili-owned lane port → TLS terminated by whichever bili answered there; hosts without the bili CA failed SELF_SIGNED_CERT_IN_CHAIN.

Fix (src/agent/native-intercept.ts only)

  • Loopback dispatches carry no hop (hoplessForLoopback): rewrite, self pass-through, post-respawn reroute/retry, and dsh-compaction divert all strip init.proxy when the final target is loopback — a loopback origin never needs a hop.
  • Fallback hop hygiene (sanitizeFallbackHop in directSendOrRefuse, the single choke point): evidence-gated per KDD-9 permitlist discipline —
    • drops an init.proxy whose normalized value matches a bili-owned origin (PI_PROXY*/BILLION_CONTEXT_PROXY env values, state.origin, state.routedOrigin, replaced origins);
    • arms a per-host NO_PROXY hold (refcounted, base restored exactly at zero, held until the awaited send settles) while the scheme-specific env hop points at a bili origin.
    • A user-chosen non-bili hop is never touched.

Behavior change (disclosure)

path before after
rewrite/self/reroute/divert to loopback host's init.proxy rode along (healthy case = self-proxy round trip; broken case = dead port) hop stripped; direct loopback dial
direct fallback host's init.proxy + env hop always honored bili-owned values dropped / scoped out via temporary NO_PROXY; user-owned values untouched

Unattributed third-party traffic (non-model URLs, passthrough) keeps its posture untouched. Node-host NO_PROXY writes are inert (undici caches env at dispatcher creation; Node fetch does not consume these vars) — Bun, the affected runtime, honors them per request.

Rebase note

Rebased onto master 23158db38; semantic conflict in directSendOrRefuse resolved as union of intent with #4516's new fallback guards (degradeLog param, local oversize refusal) — hop hygiene sits after both local-refusal paths so no hold is armed for refusals.

Tests

8 new positive+negative cases in tests/native-intercept.test.ts (#3617 gate discipline): loopback strip across the four loopback sites, fallback owned-hop drop, negative user-hop keep, env-hop NO_PROXY arming + exact base restore, concurrent-hold refcounting, retry strip. Local targeted: this file 74/74 + all nine other suites importing the module 151/151 + typecheck clean; full-suite verdict deferred to CI.

Refs #4566 (part 1 of 2).

…rt 1)

A host process memoizes its proxy hop (Bun init.proxy from PI_PROXY_ANTHROPIC /
env HTTP(S)_PROXY) for its whole life. When a lane port moves under live hosts
(ladder drift, #4566), that stale hop rides every dispatch:

- loopback-rewritten /bili/ targets get dialed through the dead port ->
  connection refused on every model turn;
- direct fallbacks keep the hop even when it points at a bili-owned lane port
  -> TLS terminates on whichever bili answers there and hosts without the
  bili CA fail SELF_SIGNED_CERT_IN_CHAIN.

src/agent/native-intercept.ts:
- hoplessForLoopback: every dispatch whose final target is loopback (rewrite,
  self pass-through, post-respawn reroute/retry, dsh-compaction divert) now
  strips init.proxy — a loopback origin never needs a hop.
- sanitizeFallbackHop in directSendOrRefuse: evidence-gated cleanup for direct
  fallbacks — drops an init.proxy whose normalized value matches a bili-owned
  origin (PI_PROXY*/BILLION_CONTEXT_PROXY env values, state.origin,
  state.routedOrigin, replaced origins), and arms a per-host NO_PROXY hold
  (refcounted, base restored exactly at zero, held until the send settles)
  when the scheme-specific env hop is bili-owned. User-chosen non-bili hops
  are never touched (KDD-9 permitlist discipline).

Behavior change (disclosure): intercepted dispatches no longer carry the
host's init.proxy to loopback targets (healthy-lane case was a harmless
self-proxy round trip before); fallback dispatches drop bili-owned hops and
scope a temporary NO_PROXY entry. Unattributed third-party traffic keeps its
posture untouched. Node-host NO_PROXY writes are inert (undici caches env at
dispatcher creation); Bun — the affected runtime — honors them per request.

Tests: 8 new positive+negative cases in tests/native-intercept.test.ts
(#3617 gate discipline), incl. concurrent-hold refcounting and exact
NO_PROXY base restoration. All 74 file-local + 150 cross-suite targeted
tests green; full-suite verdict deferred to CI.

Part 2 of #4566 (the lane itself moving under serving hosts) awaits the
owner decision requested in the issue thread.
@ranxianglei ranxianglei changed the title fix: drop stale host proxy hops from intercepted dispatches (#4566 part 1) [PR] fix: drop stale host proxy hops from intercepted dispatches (#4566 part 1) Oct 26, 2026
@github-actions

github-actions Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-25_fix-4566-stale-proxy-hop (4d4cde1)

Latest build — install this exact version

npm install -g billion-context@0.0.8796-pr.4878.49902a1

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Recent builds (commit → version)

commit version build
4d4cde1 billion-context@0.0.8796-pr.4878.49902a1 build
993958b billion-context@0.0.7175-pr.4878.7ab4b4f build
76a7df2 billion-context@0.0.6843-pr.4878.310fb07 build
ed3f171 billion-context@0.0.6353-pr.4878.adc7889 build
689844a billion-context@0.0.6322-pr.4878.7f8f084 build
28338a7 billion-context@0.0.6315-pr.4878.0f89f4d build

Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr4878.tgz
npm install -g package

Install instructions are refreshed on each push.

…+ divert-site strip (#3617)

#3617 requires both directions per gate; the loopback strip gate shipped
without its keep direction (hop must RIDE when the final target is NOT
loopback, e.g. a remote bili instance), and the dsh-compaction divert site
claimed in the PR description had no dedicated row. Added both:

- negative: non-loopback origin keeps the host's own init.proxy intact;
- positive: diverted dsh-compaction dispatch carries no stale hop.
@ranxianglei ranxianglei added bug Something isn't working p1 scope/wiring ci:full Run the FULL CI matrix (all e2e lanes + 12-leg test matrix) on this PR and removed bug Something isn't working p1 scope/wiring labels Oct 26, 2026
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

进度汇报(进程被看门狗重启前的状态)

上一条审核回复之后的进展:

  1. 发现 CI 异常并处置:我的 push(17:14Z)的 synchronize 事件没有触发任何 workflow run——该分支上当时全部 30 个 run 都是 17:26:50Z 一批 labeled 事件(镜像标签同步扇出 ×3,concurrency 组内互相取消)。而 ci.yml 的 plan job 对非 ci:full 标签事件按设计 scope=skip:幸存的那个 "CI success" run 只跑了 12 个 guard,test/static-gates/mysql-contract 全部 skipped——也就是说 head 689844a53 上全量测试矩阵从未真正跑过。不是代码问题,是触发面问题。
  2. 已用设计内机制补触发:给 PR 加了 ci:full 标签(GitHub API),17:53Z 起 full 矩阵(16 test legs + 4 static)在 689844a53 上开跑。此前各 e2e 车道(real codex / fake / persona / native-pi / native-dsh / native-opencode / pi-codex-ws / local-registry / windows codex overlay)在 17:26Z 那批 run 里已全部 success。
  3. 当前阻塞:共享账号 PAT 的 GitHub API 被并行会话限流,连续多次轮询失败(限流窗口整点重置);沙箱也出不了 github.com 网页面。full 矩阵按规模预计 20–40 分钟跑完。我会用短间隔轮询等它结束,出结果后立刻跟最终判定评论。

审核结论不变:代码与行为独立复核通过、无回归、#3617 双向已补齐;唯一待办就是这个 head 的全量 CI 绿。合并权在 owner。

@ranxianglei ranxianglei added ci:full Run the FULL CI matrix (all e2e lanes + 12-leg test matrix) on this PR and removed ci:full Run the FULL CI matrix (all e2e lanes + 12-leg test matrix) on this PR labels Oct 26, 2026
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

全量套件结论落地:CI 全绿,PR mergeable_state=clean,可合并。

接本线程前面的完整审核与进度报告。补充说明触发过程:我 17:14Z 推送补测提交后,synchronize webhook 没有送达(该 push 没有任何 workflow run 产生);随后镜像标签同步的 labeled 事件按 ci.yml(#3295)设计都是 scope=skip(非 ci:full 的 labeled 事件不跑矩阵)。于是用两条正规路径补齐了全量:

  1. workflow_dispatch(full=true)对分支 head 跑了完整矩阵(run 40381810547):16/16 test + 4/4 static-gates,零失败;
  2. 移除并重新添加 ci:full 标签,对 merge ref 触发了正式的 pull_request 全量矩阵 run(40390660397):34 jobs = 16/16 test + 4/4 static + 全部 guard 绿,仅 schedule-only 的 red-signal 跳过;9 条 e2e 车道(real/fake/persona codex、native pi/dsh/opencode、pi-codex-ws、local-registry、windows codex overlay)全部绿。

结构化八问

  1. 是什么问题:宿主进程把代理 hop(init.proxy / env)memoize 整个进程生命周期,lane 端口漂移后,intercept 执行的每次 dispatch 都带着死掉的 hop → 每个模型回合 connection refused,或 fallback 被某个 bili 终止 TLS 导致 SELF_SIGNED_CERT_IN_CHAIN。Part 1 修 hop 卫生(wiring 层);Part 2(lane 本身在活宿主脚下移动)等 [Bug][omp lane] lane port moves under live omp hosts, and the host's cached proxy hop survives re-targeting — every Anthropic turn dies (connection refused / SELF_SIGNED_CERT_IN_CHAIN) #4566 里 A1/A2 拍板后单独落。
  2. 什么原因导致的:[Bug?] Under high load, health check timeouts appear to be misinterpreted as "proxy unavailable": large session requests | [Bug?] 高负载下 health 探活超时疑似被当成「代理不可用」:大会话请求被本地拒发(0.1.199 / plugin native) #3654 让 intercept 每请求重解析 TARGET,但从不重解析 HOP——目标跟着走了,hop 还钉在宿主启动时的旧端口上。
  3. 影响范围与严重程度:p1——受影响宿主的模型流量完全不可用(每回合必败);触发条件是 lane 端口漂移(ladder 重新分配)且宿主进程存活。
  4. 怎么修复的:两层卫生,均按 KDD-9 permitlist 纪律做证据门控:loopback 终态目标一律不带 hop(hoplessForLoopback,覆盖 rewrite/self/reroute-retry/divert 四个位点);fallback 路径在单一咽喉点(directSendOrRefuse 内 sanitizeFallbackHop)丢弃 bili-owned 的 hop 值,并用引用计数、base 精确恢复的临时 NO_PROXY 作用域化 env hop。用户自选的非 bili hop 一律不动。本轮审核另补齐 Test Blind Spot Retrospective: Reinforcing Negative Assertion Discipline, Tightening Fake Upstream Strictness, Thin Inst | 测试盲区复盘:补反向断言纪律、假上游严格化、薄安装冒烟、形态矩阵与生命周期覆盖 #3617 要求的双向测试缺口(非 loopback keep 方向行 + dsh-divert 位点),commit 689844a(test-only,+27 行)。
  5. 是否有回归问题:无。触碰路径的既有行为逐项清点过:未归属第三方流量(非模型 URL、passthrough)姿态不变;Node 宿主上 NO_PROXY 写入是惰性的(undici/fetch 不消费这些变量),Bun 按请求消费——两种运行时都不产生新向量;用户自有 hop 不受影响;本地拒发路径不 arm hold。分支 head 与 merge ref 两侧全量矩阵均绿。
  6. 是否破坏缓存:不涉及。未触碰缓存/重试/增量游标/去重键任何失效策略;hop 卫生只改拨号路由选择。
  7. 是否改了设计方向:没有。留在既有 intercept dispatch 架构内;Part 2 的方向等 owner 在 [Bug][omp lane] lane port moves under live omp hosts, and the host's cached proxy hop survives re-targeting — every Anthropic turn dies (connection refused / SELF_SIGNED_CERT_IN_CHAIN) #4566 拍板。
  8. 是否引入了新的配置:无新 env/config/默认值。

判定:可合并(合并权在你,我不代合)。Part 2 待 A1/A2 决策后单独出 PR。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)
Re-synced origin/master (it had advanced another 124 commits since the last sync) into this branch — clean fast-forward push to 993958b; none of those new master commits touched this PR's files. Merge state: behind → blocked (no longer behind; 22 checks queued on the new head).

…-stale-proxy-hop

Conflict resolution (src/agent/native-intercept.ts) — both intents preserved:

- directSendOrRefuse: current master's #5405/#5408 silent-direct byte-ceiling
  refusal runs first (a refusal arms no NO_PROXY hold), then #4566's
  sanitizeFallbackHop wraps the as-is/restored sends in try/finally.
- routed / self / rewrite / retry sites: #4566's hoplessForLoopback(target, init)
  is combined with master's allExtra / branchExtra pf-watermark headers
  (#5403/#5418) and its self-branch stamp+learn.
@Maziar123

Copy link
Copy Markdown
Contributor

I prepared an additive merge candidate for this PR against current master.

Candidate commit: Maziar123@4d4cde1

It resolves the conflicts in src/agent/native-intercept.ts while preserving this PR's #4566 stale-hop sanitization and current master's byte-ceiling and watermark-header changes. Targeted verification: node --import tsx --test tests/native-intercept.test.ts passed 92/92, and npm run typecheck is clean. Three pre-existing failures remain in tests/issue3304-incremental-direct-send.test.ts and tests/opencode-native.test.ts; they reproduce unchanged on both current master and this PR head and do not touch the changed files.

I cannot push directly to this PR's branch, so the candidate is on my fork: https://github.com/Maziar123/billion-context/tree/local-repair/pr-4878-20261109. Please cherry-pick or use it as a reference.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 Merge conflict resolved & pushed (attempt 1/3, CI engineer)

I resolved the dirty merge state on 2026-10-25_fix-4566-stale-proxy-hop and fast-forward pushed it. Credit where due: @Maziar123 prepared the resolution candidate; I independently verified it line-by-line and locally, then pushed it.

What was wrong: the branch was cut off an old master (44067a5bc) and master advanced ~1005 commits past it, so src/agent/native-intercept.ts conflicted — the PR's #4566 stale-hop sanitization vs master's #5405/#5408 silent-direct byte-ceiling refusal + #5403/#5418 pf-watermark headers.

How I resolved it: adopted Maziar123's additive merge candidate 4d4cde18 (parents = [993958b795 PR-head, 406193b2 current-master]). Because it merges current master in, the branch now contains all of master → the base conflict is structurally gone. A merge (not a rebase) means a plain fast-forward push — no force-push. It changes only two files vs master: src/agent/native-intercept.ts (+269/-18) and tests/native-intercept.test.ts (+253).

Verification I ran this run (targeted, per local-testing discipline):

  • Read the full merged native-intercept.ts diff: master's byte-ceiling refusal (unattributedByteOversizeRefusal) still runs first (a refused request arms no NO_PROXY hold), then [Bug][omp lane] lane port moves under live omp hosts, and the host's cached proxy hop survives re-targeting — every Anthropic turn dies (connection refused / SELF_SIGNED_CERT_IN_CHAIN) #4566's sanitizeFallbackHop wraps the as-is/restored sends in try/finally{dispose()}; every loopback site (routed/self/rewrite/retry/dsh-divert) gets hoplessForLoopback(target,init) combined with master's pf-watermark headers; _resetForTest clears the new globals. Both intents intact.
  • npm ci → built kernel/ + pi-subagents/ → tsc --noEmit: clean.
  • tests/native-intercept.test.ts: 92/92 pass.
  • The two suites flagged as pre-existing failures (issue3304-incremental-direct-send, opencode-native): pass locally (23/23, 65/65) → supports that the 7 red shards were flaky/environmental at the old sha, not code regressions.

Push + live CI (verified against the mirror cache this run, not from memory): confirmed remote head was still 993958b795 (no concurrent agent advanced it) → fast-forward push landed: 993958b795..4d4cde186b. New head 4d4cde186b is under CI evaluation: failed 0, 17/30 green, 8 long-running lanes still in progress (the three zero-token e2e lanes, mysql-contract, docs-coverage, pi-subagents-guard, unused-exports, kernel-guard). No failures observed on the resolved head — the prior 7 shard failures did not reproduce. I have not yet observed the full run go fully green (slow lanes pending), so treat final-green as pending rather than asserted.


七问(中文结构化结论)

  1. 是什么问题:PR [PR] fix: drop stale host proxy hops from intercepted dispatches (#4566 part 1) #4878 与 base 分支冲突(merge state dirty),卡住无法合并。
  2. 什么原因导致的:分支基于旧 master(44067a5bc)切出,master 又前进了约 1005 个提交;src/agent/native-intercept.ts 被两侧同时改动——PR 侧 [Bug][omp lane] lane port moves under live omp hosts, and the host's cached proxy hop survives re-targeting — every Anthropic turn dies (connection refused / SELF_SIGNED_CERT_IN_CHAIN) #4566 stale-hop 卫生 vs master 侧 native intercept: oversized model request silently direct-sent when live-state slot is empty (975K tokens, no log) #5405/[Bug][dsh][storage] Compression permanently fails for a session: frozen ingest watermark, requests no longer observed, and no warning until the host dies with CONTEXT_WINDOW_EXCEEDED #5408 byte-ceiling 拒绝 + [Bug][dsh] 20k-message / 81 MiB-inbound session: host aborted 6× in 2 h — the #3050 resident-AST OOM is not bounded by sessions.hardMaxBytes (Windows, 0.1.210) #5403/[Bug][dsh] Plugin lane performs a full re-parse and resends the entire body every round (81 MiB → hundreds of MB AST), w | [Bug][dsh] plugin 车道每轮全量解析重发 body(81 MiB→数百 MB AST)是 #5403 OOM 的生产者:用水位线断言替换逐条复核,服务端只解析新增尾段 #5418 pf-watermark 头——形成文本冲突。
  3. 影响范围与严重程度:仅阻塞本 PR 合并,不影响任何已发布行为;不解决则 [Bug][omp lane] lane port moves under live omp hosts, and the host's cached proxy hop survives re-targeting — every Anthropic turn dies (connection refused / SELF_SIGNED_CERT_IN_CHAIN) #4566 part 1 无法落地。属「特定条件(base 落后)下有问题」,非数据/可用性缺陷。
  4. 怎么修复的:采用协作者 Maziar123 预制的 FF 合并候选(4d4cde18,parents=PR-head+当前 master);我逐行核对合并 diff 确认两侧意图都保留后,远端头仍为旧值时做 fast-forward push(无 force)。本地 typecheck 干净 + native-intercept 92/92 + 两个此前报红套件本地全绿。
  5. 是否有回归问题:无。相对 master 仅动 native-intercept.ts 与其测试两个文件;未触碰 kernel/**、pi-subagents/src/**、配置面、持久化格式、wire 协议。既有行为逐项核对保留:byte-ceiling 拒绝先于 hop 清理(拒绝不 arm NO_PROXY hold)、各 loopback 站点去 stale hop、pf-watermark 头原样、_resetForTest 重置新增全局态。
  6. 是否破坏缓存:不涉及。改动在 dispatch hop 卫生层,不改缓存/重试/增量游标/去重键语义;pf-watermark 头按原样保留。
  7. 是否引入新配置:不涉及。无新 env/config/默认值(NO_PROXY 的 scoped hold 是运行时内部机制,不是对外配置项)。

判定:冲突已解决并推送,本地定向验证全绿,CI 新 head 目前 0 失败(慢速 e2e/guard 车道仍在跑)。收工;最终全绿以 CI 为准。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 Already green — no fix needed (re-verified live this run)

Re-checked against live sources before touching anything:

  • Branch head is now 4d4cde186b — the conflict-resolving merge I pushed last turn. The 7 failing shards in this event are from run …71331 at the old, superseded head 993958b795; that commit is no longer the branch tip, so those failures are moot.
  • On the current head 4d4cde186b, the mirror cache now reports failed: 0, pending: 0 → 29 success + 1 skipped (red-signal, skipped by design) across all 30 checks. The real-client e2e lanes (opencode native, codex persona/continuity, OpenCode V2 WS), registry e2e, mysql-contract, and the guards (kernel-guard, pi-subagents-guard, docs-coverage, unused-exports) all completed green. None of the previously-red test shards fail here.

So the earlier wake already cleared what this event reports — nothing left to fix. Stopping without further code work.


七问(中文结构化结论)

  1. 是什么问题:事件报告 7 个 test 分片红(ubuntu/windows 各节点),head 标为 993958b795。
  2. 什么原因导致的:这些失败落在旧 head 993958b795 上——它基于落后的旧 master,属继承性/偶发红;上一轮我已把分支 fast-forward 合并到当前 master(4d4cde186b),该旧提交不再是分支头,红随之消失。
  3. 影响范围与严重程度:无需修复——当前 head 已全绿,不影响任何已发布或 PR 内行为。
  4. 怎么处理的:仅 live 复核,未改任何代码。先 git fetch 确认远端头=4d4cde186b,再读镜像缓存确认新 head failed:0 / pending:0(29 success + 1 skipped)。
  5. 是否有回归问题:无(本轮零改动)。
  6. 是否破坏缓存:不涉及。
  7. 是否引入新配置:不涉及。

判定:当前 head CI 全绿,冲突与红 CI 均已解决,收工,无进一步工作。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[ework] CI fix verified green ✓

@ranxianglei
ranxianglei merged commit 2cd9edb into master Nov 9, 2026
58 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working ci:full Run the FULL CI matrix (all e2e lanes + 12-leg test matrix) on this PR p1 scope/wiring

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants