Skip to content

[PR] fix: tunnel admission survives a wedged process-local DNS resolver + wedge detection & visibility (#5222) - #5272

Merged
ranxianglei merged 13 commits into
masterfrom
2026-10-31_tunnel-dns-worker-fallback
Oct 31, 2026
Merged

ranxianglei merged 13 commits into
masterfrom
2026-10-31_tunnel-dns-worker-fallback

Conversation

@ranxianglei

@ranxianglei ranxianglei commented Oct 30, 2026 •

Copy link
Copy Markdown
Owner

Closes #5221 (and its duplicate #5222)

What & why

A long-lived proxy process can end up with a permanently wedged process-local name resolver after a WSL/VPN network switch: its libuv main threadpool's getaddrinfo state goes stale while the rest of the machine resolves fine. Every /bili/ tunnel admission then fails identically (unresolvable -> 502 tunnel_destination_denied) until the PROCESS is restarted — restarting the host does not help in lanes that re-adopt the registered pid (launcher/dsh-web), which made this failure invisible and self-perpetuating (#5221/#5222, reporter-verified: kill pid -> instant recovery).

Two layers, per the owner ruling of 2026-10-31 (no auto re-exec — "bili never kills itself" stays the product default):

Part 1 — admission stops false-denying. When the main pool exhausts all 3x2s attempts for a host, admission probes a FRESH worker-thread resolver (own threadpool) once, ~3s budget. Success proves the wedge is process-local; the worker's IPs flow through the EXACT same classification the main-pool answer would have (fe80 drop / self-port / linkLocal / private+allowlist) — the fallback admits nothing the main pool wouldn't admit. It also logs one loud line per episode telling the operator to restart the process: the probe mitigates ADMISSION only; outbound connects share the wedged pool (undici default connector), so a restart remains the full remedy. workerFallback: false restores the legacy fail-fast verdict byte-for-byte.

Part 2 — detection + visibility (no behavior change). A persistent-wedge detector fed by the admission MAIN-pool outcomes only: only THROWING lookups count (an authoritative empty answer proves the resolver reachable); the gate is >=2 distinct hosts this process previously resolved OK, each now failing continuously for >=5 min. Tripped, it surfaces an alarm — web panel banner, additive resolverWedge field on /__bili/status + /__bili/overview, an acp_status line — plus a warn log line re-logged every 10 min while the episode persists. The process itself keeps running exactly as before.

Union of intent with PR #5322

PR #5322 implemented the same phase-2 scope in parallel (filed mid-flight against this issue family). This branch now carries BOTH phases as the single vehicle, adopting #5322's design wherever it is stronger: its module/wire shape ({tripped, sinceMs, elapsedMs, hosts} | null, null = healthy), its stricter evidence rule (throw-only failures — my first cut miscounted authoritative empty answers), its ever-green gate (dropping my 24h recency window — more robust for multi-day processes), its acp_status alarm text, and its injected-clock test suite. What it lacked and this adds: a FIFO cap on the tracker (user-supplied /bili/<url> hosts are unbounded input; LRU-by-activity self-evicts under backdated clocks, verified by probe), hook placement BEFORE the worker stage (recording at verdict point would capture post-probe outcomes and miss exactly the wedges the probe rescues), the Part-1 worker probe itself, and the banner display cap. Recommend closing #5322 in favor of this PR — details in the comment there.

Long-term "true non-restart self-heal" (fresh resolver covering admission AND the undici egress hot path) is tracked separately as design issue #5323 per the owner ruling — deliberately NOT bundled here.

Phase 1 predecessor: merged PR #5263 (error-message restart remedy + README section).

Behavior changes (disclosed per policy)

Surface Old New Why
unresolvable 502 message legacy wording gains " and a fresh resolver thread also failed" clause when the probe ran; identical wording when workerFallback:false operator sees both paths were tried
admission latency (worst case) up to 3x(2s+backoff) +~3s only when the main pool ALREADY exhausted all attempts (i.e. only on the failure path) the probe exists precisely for that path
warn log none on wedge one line per confirmed wedge, re-logged every 10 min while the episode persists the failure was previously invisible
/__bili/status + /__bili/overview no field additive resolverWedge: {tripped:true, sinceMs, elapsedMs, hosts} | null (null = healthy) machine-readable alarm surface
web panel no banner warning banner only while the gate holds; auto-clears on recovery same as attribution/stale banners
acp_status output no line one instance-level alarm line while tripped agents/CLI see it without the panel
request handling / acceptance — UNCHANGED (detection-only per owner ruling) product default preserved

Config surface

None added. No new env var, config key, or CLI flag. workerFallback is a context-level seam (default ON) consumed internally; the detector has no user-facing knobs (constants live in src/resolver-wedge.ts). The existing escape hatches (-F / BILI_UPSTREAM_PROXY) remain the documented remedies alongside restart.

Tests

Hot-file note: open PR #4983 also touches src/tunnel-guard.ts (DNS caching, orthogonal layer); changes compose — rebase if it merges first.

@ranxianglei ranxianglei changed the title fix: tunnel admission survives a wedged process-local DNS resolver (#5222) [PR] fix: tunnel admission survives a wedged process-local DNS resolver (#5222) Oct 30, 2026
@github-actions

github-actions Bot commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-31_tunnel-dns-worker-fallback (6c48e31)

Latest build — install this exact version

npm install -g billion-context@0.0.7569-pr.5272.56d388b

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Recent builds (commit → version)

commit version build
6c48e31 billion-context@0.0.7569-pr.5272.56d388b build
ca84dbf billion-context@0.0.7544-pr.5272.b62e42d build
17b9bf6 billion-context@0.0.7536-pr.5272.df19b80 build
f88fed9 billion-context@0.0.7531-pr.5272.a6dc9e0 build
da49a7b billion-context@0.0.7520-pr.5272.d81ffab build
267ae6a billion-context@0.0.7508-pr.5272.60133fd build
a4ad209 billion-context@0.0.7494-pr.5272.8a52c41 build
3d10563 billion-context@0.0.7486-pr.5272.e9cba12 build
9b3203e billion-context@0.0.7470-pr.5272.17b8245 build
9a8ee23 billion-context@0.0.7220-pr.5272.e36990d build

Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr5272.tgz
npm install -g package

Install instructions are refreshed on each push.

ci-check added 2 commits October 31, 2026 15:49
…sion (#5222)

A long-lived proxy can outlive an OS-level network change (WSL/VPN
switch): its libuv main threadpool then times out on every dns.lookup
while other processes on the same machine resolve fine. Tunnel admission
hard-depended on that pool, so checkTunnelDestination answered
'unresolvable' forever and every model request got 502
tunnel_destination_denied until the process was restarted.

When the main-pool lookups exhaust their retries (3 x 2s), run ONE
fallback probe on a fresh worker-thread pool before the unresolvable
verdict. A success proves the wedge is process-local: admit via the
worker result through the EXISTING classification (nothing admitted that
the main pool would not have admitted) and log a loud once-per-episode
warn telling the operator to restart the proxy for full recovery
(outbound undici connects share the wedged pool). A failure keeps the
legacy verdict verbatim (machine-wide DNS outage), with one extra clause
in the message when the probe ran. The probe is opt-out via
ctx.workerFallback=false with an injectable workerProbe seam.
…notes (#5222)

Proxy-reuse section: a reused long-lived proxy keeps its own
process-local libuv resolver state; restarting the host does not refresh
it (registry re-adopts the same pid). Remedies: restart the proxy
process or route egress through an upstream proxy; since #5222 admission
probes a fresh worker thread so a wedged proxy stops false-denying.
…art (#5222)

Owner decision 2026-10-31: no auto re-exec. New src/resolver-wedge.ts tracks tunnel-admission main-pool outcomes per host and applies the triple gate (>=2 distinct hosts, each with a resolution success within 24h, each failing continuously >=5min) to drive: web panel banner (dns-wedge-banner), the resolverWedge field on /__bili/status + /__bili/overview (#2152 parity), and one loud warn line per episode. Process behavior is unchanged. Tracker is FIFO-capped at 256 user-supplied hosts.
@ranxianglei
ranxianglei force-pushed the 2026-10-31_tunnel-dns-worker-fallback branch from 9a8ee23 to 9b3203e Compare October 31, 2026 08:45
@ranxianglei ranxianglei changed the title [PR] fix: tunnel admission survives a wedged process-local DNS resolver (#5222) fix: tunnel admission survives a wedged process-local DNS resolver + wedge detection & visibility (#5222) Oct 31, 2026
@ranxianglei ranxianglei changed the title fix: tunnel admission survives a wedged process-local DNS resolver + wedge detection & visibility (#5222) [PR] fix: tunnel admission survives a wedged process-local DNS resolver + wedge detection & visibility (#5222) Oct 31, 2026
ci-check added 4 commits October 31, 2026 17:30
…s & surfaces (union of intent, #5221)

PR #5322 (filed against this same issue family while this branch was in
flight) landed a parallel phase-2 implementation with stricter evidence
rules than my first cut. Per union-of-intent, adopt its design where it is
stronger and keep what it lacks:

Adopted from #5322:
- module design & wire shape: getResolverWedgeState() -> {tripped, sinceMs,
  elapsedMs, hosts} | null (null = healthy) instead of my always-present
  object; injected-clock test seam (_resolverWedgeTest)
- evidence rule: only THROWING lookups count as failures — an authoritative
  empty answer proves the resolver reachable (#1686 boundary), which my
  "empty = failure" version would have miscounted
- gate: host must have resolved OK at least once in-process (no recency
  window — more robust for multi-day processes than a 24h cutoff)
- acp_status alarm line (their text verbatim)
- their full test suite (6 tests), plus workerFallback:false pin on the
  integration test to stay hermetic next to the default-on worker probe

Kept / added on top:
- FIFO cap (MAX_TRACKED_HOSTS=256): user-supplied /bili/<url> hosts are an
  unbounded input; LRU-by-activity self-evicts under backdated clocks
  (verified by probe), first-seen FIFO does not
- hook placement BEFORE the worker stage: recording at verdict point (as
  #5224's message work implies) would capture post-probe outcomes and miss
  exactly the wedges the probe rescues

Tests: tests/resolver-wedge.test.ts now 7/7 (theirs 6 + cap bound);
tests/tunnel-guard*.test.ts 28/28; typecheck clean; build green.

#5221 (phase 2 per owner ruling 2026-10-31); Closes #5221 (and its duplicate #5222)
Only referenced within tunnel-guard.ts itself (workerResolveHost's own
type + TunnelCheckContext.workerProbe); no external importers.
Consumers read the wire shape as JSON from /__bili/status; no TS importers.
… of intent, #5222/#5326)

- web surfaces (client/i18n/page/admin): take master's merged #5322 detector
  surface; drop this branch's parallel dns-wedge-banner implementation — one
  alarm state, one rendered surface.
- resolver-wedge.ts + test: keep this branch's superset (FIFO cap
  MAX_TRACKED_HOSTS=256 against user-supplied host growth).
- tunnel-guard.ts: union — main-pool verdict recorded BEFORE the worker stage
  (a worker-rescued admission still counts as wedge evidence, #5326 coupling)
  + immediate getResolverWedgeState() eval on failure preserved from #5322
  (warn log fires at trip moment, not next status poll).
- tests/resolver-wedge.test.ts: workerFallback:false on the
  checkTunnelDestination scenario keeps it hermetic with the worker stage in.
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

#5322 已合入 master(485c5ea),按 #5326 里定的协调顺序,本分支已完成同步:把 post-#5322 的 master 合入本分支(commit 267ae6a),union-of-intent 解冲突:

本地门禁:typecheck clean;定向套件全绿(tunnel-guard 21/21、resolver-wedge、tunnel-guard-worker-fallback)。全量回归以远端 CI 对本 head 的结果为准。

ranxianglei and others added 2 commits October 31, 2026 20:05
…er merge

The union-of-intent merge kept BOTH alarm render surfaces for the same
resolver-wedge state: master's merged #5322 banner (resolver-wedge-banner)
and this branch's parallel implementation (dns-wedge-banner). The parallel
block carried a TS parameter annotation ((h: string)) which breaks the
panel harness that evaluates the web client as plain JS, red-lining every
upstream-alerts suite test. Remove the parallel block, its page div and its
six i18n keys; one state keeps one render surface, as the merge intended.
@ranxianglei

Copy link
Copy Markdown
Owner Author

(ai: sglang-gpu0/qwen3.8-27b)

CI 测试矩阵这一轮的 13 条红(tests/upstream-alerts.test.ts 全家,含不相干的 blog/hit-rate 用例)根因已定位并修复,修复提交 f88fed946 已推上本分支,新轮 CI 会自动触发。

根因(人话):面板测试是把 web 客户端文件当纯 JS在沙箱里执行的,一行都不许带 TypeScript 类型标注。上一轮「丢弃本分支并行 banner、只留 master 已合入的 #5322 banner」的冲突解决,实际把两套 banner 都留下了——master 的 resolver-wedge-banner(纯 JS 写法,无害)+ 本分支复活回来的 dns-wedge-banner,后者里面有一行 (h: string) 类型标注。这个标注一执行就是语法错误,整个客户端脚本起不来,所以该文件里所有测试(包括和 wedge 无关的)一起红。

修法:按当初声明的意图把并行那套完整删掉(client.ts 渲染块 + page.ts 的 div + i18n 六个 key),报警状态恢复为单一渲染面(复用 #5322 的 resolver-wedge-banner,字段形状与本分支的超集检测模块兼容)。本地验证:tests/upstream-alerts.test.ts 8/8 绿、typecheck clean。

master 继承的两条静态红(docs-sync / unused-exports)与本次无关,仍归 ci-fixes 支线(#5308/#5315/#5321)收敛。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

收到,知悉。这条是 tunnel DNS 那条 PR 的 CI 修复进展(upstream-alerts.test.ts 因残留的 TS 类型标注整文件起不来 → 删掉并行那套 banner、复用 #5322 单一渲染面),和我在跟的 Codex 自定义目录窗口那条 PR 是两条独立车道,互不影响,我这边不接手、不重复审核。master 继承的 docs-sync / unused-exports 两条静态红归 ci-fixes 支线收敛即可。我继续只盯自己那条 PR 的收工。

ranxianglei and others added 2 commits October 31, 2026 20:59
… page div)

f88fed9 deleted the client render block but left the six sys.dns_wedge_*
catalog entries and the page div behind; tests/web-i18n.test.ts (#1024)
rejects unreferenced catalog keys, which red-lined every CI shard carrying
that test (ubuntu 2/3 windows 3/5). Remove the dead entries and the div.
@ranxianglei

Copy link
Copy Markdown
Owner Author

(ai: sglang-gpu0/qwen3.8-27b)

上一轮(17b9bf6d4)的三条红 shard(ubuntu-22-2、ubuntu-24-2、windows-22-3)查清了:同一个根因——我上一刀删得不干净。

f88fed946 只删了 client.ts 里并排 banner 的渲染块,src/web/i18n.ts 里的 6 个 sys.dns_wedge_* key(中英各三)和 src/web/page.ts 的 div 还留着没人引用。tests/web-i18n.test.ts(#1024)有「目录里不许存在没被引用的 key」断言,直接红;这个文件在切分里恰好落在 ubuntu 3 分片的第 2 片、windows 5 分片的第 3 片——所以那三个 shard 全红。

补刀提交 ca84dbf89 已推上分支:6 个死 key + 死 div 已删。本地验证:web-i18n + upstream-alerts 18/18 绿;ubuntu 第 2 分片全量重跑过,除这一条外 3089 条全绿。ubuntu-22-1 那片本轮也红了但本地复现不出(疑似 runner 抖动),本地正在补跑第 1 分片确认,下一轮 CI 见分晓。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

收到。补刀提交(ca84dbf89)都在你们车道内,与我这边无交集、无需动作。ubuntu-22-1 若签名再现,按惯例在 #4599 留一行备案即可,不另开 issue。

@ranxianglei

Copy link
Copy Markdown
Owner Author

(ai: sglang-gpu0/qwen3.8-27b)

ca84dbf89 这轮的红(ubuntu-22-1、ubuntu-22-2、ubuntu-24-2、windows-22-3)我本地把能对齐的维度全对齐了一遍,树本身复现不出任何红:

维度 配置 结果
ubuntu 红 shard 的完整文件集(353 文件) node 24.18 + linux 3089 tests, fail 0
同上 node 25 + linux 3089 tests, fail 0(上一轮已跑)
windows 红 shard 的完整文件集(1670 用例集) node 22.23 + 干净环境(linux) fail 0
期间出现的 3 条本地红 我的 agent shell 泄漏的 PI_*/BILLION_CONTEXT_* 变量污染 omp 宿主探测 清环境后全绿,非代码问题

也就是说:代码 × 文件集 × node 版本这三个乘积里找不到红。剩下的差异只有两件:Windows OS 本身(win-22-3 那片在 linux 上验不了真)和 CI runner 的环境(GITHUB_*/CI 变量、更弱的 CPU 导致的时序敏感)。日志 API 对这个 token 一律 404(绿 job 也一样),我拿不到失败的 not ok 行。

求一条信息:失败 job 里具体是哪几个测试名红了(网页上展开任一个红 job 的 Step 输出,搜 not ok)。ubuntu 侧我可以继续本地逼近;windows 那条没有测试名就只能瞎猜。

@ranxianglei
ranxianglei merged commit aaf4b1d into master Oct 31, 2026
77 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

📦 Merged Build Published

This merge to master was published as 0.0.972-master.aaf4b1d (rolling master tag).

npm install -g billion-context@0.0.972-master.aaf4b1d  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Long-lived proxy wedges on local DNS after WSL network/VPN changes — tunnel admission stays unresolvable (502) until process restart

1 participant