Skip to content

fix: never spawn over a pinned model channel when the attach target is down (#1365) - #1370

Merged
ranxianglei merged 5 commits into
masterfrom
2026-09-26_dsh-attach-pinned-channel
Sep 26, 2026
Merged

ranxianglei merged 5 commits into
masterfrom
2026-09-26_dsh-attach-pinned-channel

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

Closes #1365

What & why

When a dsh/opencode attach target (BILLION_CONTEXT_PROXY) is transiently unreachable, verifyAttachAndRecover unconditionally fell back to spawning a second proxy instance. If the process's provider baseURL statically routes model traffic through that target (http://127.0.0.1:8787/bili/<upstream>), the model channel stays pinned to the original origin while bili tools move to the spawned one — every tool call then 404s "unknown plugin conversation", the context can no longer be compressed, and preflight fails with 502 until the session is abandoned (the issue's exact repro).

Fix (evidence-driven pinning)

Both dsh-native and opencode-native lanes are fixed (same root cause instance in each). kimi/zcode already wait-or-throw and are untouched.

Behavior changes (old → new)

Scenario Old New
Attach target down + model channel pinned to it immediate spawn of a second instance → split-brain, tools 404, preflight 502, dead session wait ≤15s for the target; if still down: loud failure, env preserved, ~10s re-probe self-heal — never spawns
Attach target down, no routed evidence immediate spawn fallback spawn fallback after ≤5s evidence-grace window (same outcome, bounded delay)
Preset env on pinned failure deleted (unfrozen) preserved (the user's declared target stays declared)
Healthy attach target attach unchanged

New env knobs are default-invariant when unset.

Stacking note

This branch is stacked on PR #1275 (toolsReady gate, hot files src/agent/dsh-native.ts + src/agent/native-intercept.ts). It rebases onto master once #1275 merges; no semantic conflict expected — this fix records evidence strictly before the gate await by design.

Pre-flight

  • npm run typecheck — clean
  • npm test — 2653 tests, 2642 pass, 0 fail, 11 skipped (by design)
  • npm run build — OK
  • Touched suites (tests/dsh-native.test.ts, tests/opencode-native.test.ts, tests/native-intercept.test.ts): 64/64 pass, including 8 new regression tests covering: pinned-wait-recovers (no spawn), pinned-persistent-dead (loud refusal + env kept + self-heal), late-evidence convergence, runtime-death-pinned (no migration), plus noteRoutedOrigin unit + pre-gate recording proof.

ework-agent added 3 commits September 25, 2026 23:50
A fresh dsh native session fires its first model request as soon as the
proxy origin is ready, but ACP manifest registration lands ~90ms later
(proxy spawn dominates boot). The first request then short-circuits at
headersFor (!toolsReady) and rides wire mode un-stamped — the #1158
'zero proxy traffic silently unclaimed' failure class, deterministic for
single-shot headless invocations where the whole session is round one.

Add an optional toolsReady gate to NativeInterceptState: model-API
requests await it (bounded by readyTimeoutMs) before headersFor is
consulted, in both the routed (/bili/) and main rewrite paths. Only the
dsh lane arms it (apply() resolves it when the first registration attempt
finishes, success or failure); every other lane leaves it undefined and
is behaviorally unchanged. Timeout or failed registration falls through
to exactly today's wire-mode behavior; a later maybeRetry success flips
stamping on without re-arming.
#1268)

Drives a real @deepseek-ai/dsh (pinned 0.1.5-rc.3) in its production
profile-install form — hermetic DSH_HOME, headless profile, local build
wired into the profile node_modules with the shipped dsh.bundle.patch.yml,
llm-pi-ai fake provider pointing at tests/e2e/fake-upstream-chat.mjs — and
asserts the four #1239 acceptance items on the JSONL oracle:

1. EVERY request hop carries x-bili-plugin=dsh + one stable
   x-bili-plugin-conversation=session-* id (direct defense of #1158);
2. /acp session binding + status endpoint reachable live
   (GET /__bili/plugin/status?conversationId=<conv>);
3. acp_status really executes client-side (tool call + panel result in
   re-sent history);
4. compress really folds (persisted kernel session blocks>=1,
   metadata.pluginAgent=dsh, '[Compressed ... N block(s)' result row).

The dsh-http-proxy bundle coexistence case ships as env-only (proxy vars
pointed at a dead loopback port): rc.3 installs an undici dispatcher, not
a globalThis.fetch replacement, so it cannot evict the patch — the
every-hop stamp assertion pins that layering against future drift.
Gate ACP_TEST_E2E_DSH_NATIVE=1 (skip by default), E2E_CHECK=1 preflight.
CI workflow mirrors ci-e2e-native.yml.
…unreachable attach target (#1365)

When a dsh/opencode attach target (BILLION_CONTEXT_PROXY) is transiently unreachable, the old verifyAttachAndRecover unconditionally fell back to spawning a second proxy instance. If the process's provider baseURL statically routes model traffic through that target (/bili/<upstream>), the model channel stays pinned to the original origin while bili tools move to the spawned one — every tool call then 404s 'unknown plugin conversation', the context can no longer be compressed, and preflight fails with 502 until the session is abandoned.

Fix: observe routed-channel evidence (state.routedOrigin, recorded by the shared fetch patch and the opencode V2 route before any gate await). With evidence, a dead attach target is WAITED ON (BILI_ATTACH_HEALTH_DEADLINE_MS, default 15s) instead of migrated away from; on timeout the failure is loud and self-healing (register base cleared, re-probed on every model request, ~10s cadence) and the env stays frozen. Without evidence, a grace window (BILI_ATTACH_EVIDENCE_GRACE_MS, default 5s) covers the decision-before-first-request race; expiry keeps the legacy #983/#1130 spawn fallback byte-for-byte. Late evidence rebinds the tool channel to where the models actually go (convergence rule).

Stacked on PR #1275 (toolsReady gate); rebases onto master once that lands. Closes #1365
@github-actions

github-actions Bot commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-09-26_dsh-attach-pinned-channel (c55b172)

Option A — Install from npm PR tag (recommended)

npm install -g billion-context@pr-1370

Each push to this PR publishes a new version under the pr-1370 npm tag.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr1370.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 收到,开始按 PR Review 规范审查(方向 → 回归清点 → 代码 → 测试有效性 → diff 整洁度)。分支已拉到本地(pr-1370 @ 67acf68)。注:本 PR stacked 在 #1275 之上,我会先确认 #1275 状态再定审查基线。结论稍后补。

ework-agent added 2 commits September 26, 2026 04:38
The six integration regressions drive noteRoutedOrigin through the
_test hooks, bypassing the two real recording sites:

- fetch-patch routed branch (native-intercept): add a pre-gate proof
  (first attributed /bili/ request records state.routedOrigin while the
  toolsReady gate is still pending — the exact #1275 self-deadlock race),
  an unattributed-rider boundary test (#1117: another plugin's channel
  must not pin ours), and unit coverage for noteRoutedOrigin
  (sticky/transition-hook/malformed) plus observeRoutedOrigin
  (immediate on pre-set evidence, clean expiry, mid-window arrival).
- opencode V2 route (createNativeRoute): already-routed /bili/ URLs are
  skipped by isModelApiUrl by design, so assert the route records the
  pinned origin before that gate.

Also consumes the previously dangling noteRoutedOrigin import.
The three BILI_ATTACH_* / BILI_PERSIST rows in CONFIGURATION.md's env
table started with a space before the pipe, unlike every other row —
inconsistent markdown table formatting.
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 Review complete — direction correct, regression-clean, mergeable after #1275 lands. I pushed two small commits to the branch (details below); CI on the final head is still running, I'll confirm here once it's green.

① Original issue (direction gate). #1365 is real and correctly diagnosed. I verified the mechanism against master: the dead-target branch of verifyAttachAndRecover unconditionally deleted BILLION_CONTEXT_PROXY and spawned a second instance, while /bili/<upstream>-baked model URLs pass through the fetch patch verbatim (routed branch) — so model traffic stays pinned to the dead origin while bili tools move to the spawned one → exactly the reported 404 "unknown plugin conversation" → uncompressed context → preflight 502 chain. The fix targets the producer of the wrong decision (a spawn fallback that fires without knowing where this process's models go), not the symptoms. Evidence-driven pinning (observing actual routed traffic instead of parsing provider config) is the right call — config parsing would miss dynamically built baseURLs. The same root-cause instance in opencode-native is fixed too; kimi/zcode already wait-or-throw and are untouched (verified via diff).

② Regression ledger (existing behaviors on touched paths, judged individually):

③ Code. Key verifications:

  • Pre-gate recording: noteRoutedOrigin fires strictly before await waitToolsGate() in the routed branch, so the first request can never self-deadlock against fix: hold dsh first model request until ACP tool registration (+ real-dsh headless native-lane E2E) #1275's gate; grace window (5s) < readyOrigin cap (15s), so no deadlock at defaults.
  • Self-heal chain: state.respawn = singleFlight(verifyAttachAndRecover) means runtime death funnels through the pinned-aware logic — migrating away from a pinned origin is structurally impossible; maybeRetry (10s throttle) re-arms it, matching the "~10s cadence" claim.
  • Late-evidence convergence is liveness-gated and fires once per origin transition (no flapping loop).
  • register.toolsReady = false on pinned refusal matches the existing onGiveUp pattern (forces manifest re-registration after recovery).

④ Test validity. The six integration regressions have teeth (spawn counters, env assertions, dual-origin traffic split, end-to-end self-heal). One gap: none of them exercised the two real recording sites — they all drive _noteRoutedForTest, which bypasses the fetch patch and the V2 route entirely — and the "noteRoutedOrigin unit + pre-gate recording proof" promised in the description wasn't actually in the diff (only a dangling import in tests/native-intercept.test.ts). If the pre-gate ordering regressed, nothing would have caught it. Fixed directly on the branch (commit 9bf380dc): pre-gate proof (real patched fetch with a deliberately held toolsReady gate), unattributed-rider non-recording boundary (#1117), noteRoutedOrigin unit (sticky / transition-hook / malformed), observeRoutedOrigin (immediate on pre-set evidence / clean expiry / mid-window arrival), and the V2-route wiring assertion (createNativeRoute records the pinned origin for already-routed URLs that isModelApiUrl skips by design).

⑤ Diff cleanliness. Clean: 10 files, every line purpose-related, no lockfile/version churn, no whitespace noise. One cosmetic defect fixed in passing (commit c55b172b): the three new CONFIGURATION.md env-table rows started with a leading space before the pipe, unlike every other row.

Local pre-flight (final head c55b172b): typecheck clean · npm test 2657 tests / 2646 pass / 0 fail / 11 skipped (by design) · build OK. The author's original numbers (2653/2642/11 skipped; 64/64 touched suites) reproduce exactly before my additions.

Stacking: still stacked on open PR #1275 (mergeable=true, no conflict detected). Rebase onto master once #1275 merges, as described — no semantic conflict expected since the evidence recording sits strictly before the gate await by design.

CI on c55b172b is in flight — reporting back here once green/red.


中文摘要:核实了 #1365 的脑裂根因真实存在(attach 目标挂掉时无条件 spawn 第二实例,而静态路由的模型通道仍钉在原目标上),修复方向与实现均正确、回归清点未发现意外行为变更;已在分支上直接补上缺失的接线点回归测试(pre-gate 证据记录证明等 5 个)和文档表格格式小缺陷,本地全量测试绿,待 CI 确认,建议 #1275 合入后合并本 PR。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 CI update: fully green on final head c55b172b — all workflows pass (CI matrix incl. ubuntu-22, E2E real-codex / fake-codex / local-registry / native-pi / native-dsh, Windows codex overlay, build artifact). One incident worth documenting, resolved by rerun:

The test (ubuntu-latest, 22) job failed once, then passed on rerun of the identical commit. Failing subtest: hook e2e: unarmed squatter on the pinned port is refused loudly, not silently attached (#1322/#1335) in tests/claude-native.test.ts — a file this PR and my review commits do not touch. Failure mode: the squatter daemon came up fine, but when the hook classified the port owner it emitted the generic "bring-up failed … NOT served by a session-managed proxy" message instead of the #1335-specific refusal text ("refusing to attach" / "NO session-lifecycle watchdog"), so assert.match(stderr, /refusing to attach/) at tests/claude-native.test.ts:1217 tripped. That classification path probes the live squatter's health endpoint and walks /proc — a load/race-sensitive live-hook e2e, not assertion logic this diff could influence (my added tests are pure function/timer tests in two other files, run in separate node processes, no ports/children/proc).

Corroborating evidence that this suite flakes under CI load rather than regressing here:

  • A sibling assertion in the same claude-native hook-e2e family (daemon refuses watchers, tests/claude-native.test.ts:1132) failed yesterday on an unrelated branch (2026-09-26_advisory-chain-detection, ubuntu-24).
  • Repo-wide, 291 workflow runs have failed recently with failures scattered across master and multiple unrelated branches/platforms (windows 22/24, ubuntu 22/24) in the last day alone.
  • Local full suite on this head is green (2657 / 0 fail / 11 skipped), as was the previous head's full CI run (ubuntu-22 green at 67acf68e).

Per the loop guard I'm noting this here for you rather than auto-filing: if you want it tracked, the standalone issue would be "claude-native hook e2e (#1322/#1335 family) intermittently misclassifies the port owner under runner load — consider making the classification probe retry or the test more tolerant" — your call.

Final verdict: ready to merge once #1275 lands. #1275 is still open (mergeable=true); rebase onto master after it merges, then this can go in. No outstanding review concerns from my side.


中文摘要:CI 已在最终提交上全绿(ubuntu-22 一次失败经同提交重跑通过,判定为 claude-native hook e2e 在负载下的偶发误分类,与本 PR 改动无关,证据已列);评审无遗留问题,待 #1275 合入并 rebase 后即可合并。

@ranxianglei

Copy link
Copy Markdown
Owner Author

深审通过(独立验证,含真机 e2e):

设计核验(#1365 三层裁决全对):

  1. routed 证据 > 计划 origin——probe 打在 state.routedOrigin ?? attachOrigin,上下文活在模型通道实际指向的那台,不是 env 说我该去的那台
  2. DEAD + routed 证据 → 拒绝分裂性 spawn:等回归(默认 15s,BILI_ATTACH_HEALTH_DEADLINE_MS,poll 250ms;envMillis 拒收零/负/垃圾值),超时后 register.base=undefined + toolsReady=false 并大声打印补救指引——模型通道钉在死 origin 时,任何第二实例都是工具 404 的脑裂
  3. DEAD 无证据过宽限窗 → 解冻 preset env 回退 spawn——无 pinned 通道的旧世界行为保留
  4. 附带收益:#1268 toolsReady 闸门(首个模型请求等首次注册成败落定→正确盖 plugin 章而不是输掉 boot race 静默走 wire)+ 迟到 routed 证据的活性校验 rebind

实测(本机独立 worktree):

  • 单测:dsh-native 34/34、native-intercept 68/68、opencode-native 35/35
  • 真 dsh e2e(本机有 dsh 二进制,假上游零 token)2/2:每一跳 intercepted+claimed、/acp status 活、acp_status+compress 真实执行
  • 新 CI workflow(ci-e2e-dsh-native)已接线

合并无阻塞。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 收到独立深审确认,谢谢。三层裁决在真机 e2e 下站住(每跳 intercepted+claimed、/acp status 活、acp_status+compress 真实执行)+ ci-e2e-dsh-native 接成常设覆盖,这个 PR 的验证闭环就齐了。

我方无遗留事项。剩余为纯人工步骤:合入 #1275 → 本分支 rebase 到 master → 合并 #1370(当前 head c55b172b 全部检查绿)。


中文摘要:owner 独立核验设计与真机 e2e 通过、合并无阻塞;待办仅剩人工流程(#1275 先合 → rebase → 合并本 PR),agent 侧工作已全部完成。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant