Repository navigation
[PR] fix: tunnel admission survives a wedged process-local DNS resolver + wedge detection & visibility (#5222) - #5272
Conversation
📦 Built Package ArtifactBranch: Latest build — install this exact versionnpm install -g billion-context@0.0.7569-pr.5272.56d388bThis exact version is immutable and reproducible. The rolling Recent builds (commit → version)
Download artifact
tar xzf billion-context-pr5272.tgz
npm install -g packageInstall instructions are refreshed on each push. |
…sion (#5222) A long-lived proxy can outlive an OS-level network change (WSL/VPN switch): its libuv main threadpool then times out on every dns.lookup while other processes on the same machine resolve fine. Tunnel admission hard-depended on that pool, so checkTunnelDestination answered 'unresolvable' forever and every model request got 502 tunnel_destination_denied until the process was restarted. When the main-pool lookups exhaust their retries (3 x 2s), run ONE fallback probe on a fresh worker-thread pool before the unresolvable verdict. A success proves the wedge is process-local: admit via the worker result through the EXISTING classification (nothing admitted that the main pool would not have admitted) and log a loud once-per-episode warn telling the operator to restart the proxy for full recovery (outbound undici connects share the wedged pool). A failure keeps the legacy verdict verbatim (machine-wide DNS outage), with one extra clause in the message when the probe ran. The probe is opt-out via ctx.workerFallback=false with an injectable workerProbe seam.
…notes (#5222) Proxy-reuse section: a reused long-lived proxy keeps its own process-local libuv resolver state; restarting the host does not refresh it (registry re-adopts the same pid). Remedies: restart the proxy process or route egress through an upstream proxy; since #5222 admission probes a fresh worker thread so a wedged proxy stops false-denying.
…art (#5222) Owner decision 2026-10-31: no auto re-exec. New src/resolver-wedge.ts tracks tunnel-admission main-pool outcomes per host and applies the triple gate (>=2 distinct hosts, each with a resolution success within 24h, each failing continuously >=5min) to drive: web panel banner (dns-wedge-banner), the resolverWedge field on /__bili/status + /__bili/overview (#2152 parity), and one loud warn line per episode. Process behavior is unchanged. Tracker is FIFO-capped at 256 user-supplied hosts.
9a8ee23 to
9b3203e
Compare
…s & surfaces (union of intent, #5221) PR #5322 (filed against this same issue family while this branch was in flight) landed a parallel phase-2 implementation with stricter evidence rules than my first cut. Per union-of-intent, adopt its design where it is stronger and keep what it lacks: Adopted from #5322: - module design & wire shape: getResolverWedgeState() -> {tripped, sinceMs, elapsedMs, hosts} | null (null = healthy) instead of my always-present object; injected-clock test seam (_resolverWedgeTest) - evidence rule: only THROWING lookups count as failures — an authoritative empty answer proves the resolver reachable (#1686 boundary), which my "empty = failure" version would have miscounted - gate: host must have resolved OK at least once in-process (no recency window — more robust for multi-day processes than a 24h cutoff) - acp_status alarm line (their text verbatim) - their full test suite (6 tests), plus workerFallback:false pin on the integration test to stay hermetic next to the default-on worker probe Kept / added on top: - FIFO cap (MAX_TRACKED_HOSTS=256): user-supplied /bili/<url> hosts are an unbounded input; LRU-by-activity self-evicts under backdated clocks (verified by probe), first-seen FIFO does not - hook placement BEFORE the worker stage: recording at verdict point (as #5224's message work implies) would capture post-probe outcomes and miss exactly the wedges the probe rescues Tests: tests/resolver-wedge.test.ts now 7/7 (theirs 6 + cap bound); tests/tunnel-guard*.test.ts 28/28; typecheck clean; build green. #5221 (phase 2 per owner ruling 2026-10-31); Closes #5221 (and its duplicate #5222)
Only referenced within tunnel-guard.ts itself (workerResolveHost's own type + TunnelCheckContext.workerProbe); no external importers.
…-dns-worker-fallback
Consumers read the wire shape as JSON from /__bili/status; no TS importers.
… of intent, #5222/#5326) - web surfaces (client/i18n/page/admin): take master's merged #5322 detector surface; drop this branch's parallel dns-wedge-banner implementation — one alarm state, one rendered surface. - resolver-wedge.ts + test: keep this branch's superset (FIFO cap MAX_TRACKED_HOSTS=256 against user-supplied host growth). - tunnel-guard.ts: union — main-pool verdict recorded BEFORE the worker stage (a worker-rescued admission still counts as wedge evidence, #5326 coupling) + immediate getResolverWedgeState() eval on failure preserved from #5322 (warn log fires at trip moment, not next status poll). - tests/resolver-wedge.test.ts: workerFallback:false on the checkTunnelDestination scenario keeps it hermetic with the worker stage in.
(ai: vllm-qwen/qwen3.8-27b) #5322 已合入 master(485c5ea),按 #5326 里定的协调顺序,本分支已完成同步:把 post-#5322 的 master 合入本分支(commit 267ae6a),union-of-intent 解冲突:
本地门禁:typecheck clean;定向套件全绿(tunnel-guard 21/21、resolver-wedge、tunnel-guard-worker-fallback)。全量回归以远端 CI 对本 head 的结果为准。 |
…er merge The union-of-intent merge kept BOTH alarm render surfaces for the same resolver-wedge state: master's merged #5322 banner (resolver-wedge-banner) and this branch's parallel implementation (dns-wedge-banner). The parallel block carried a TS parameter annotation ((h: string)) which breaks the panel harness that evaluates the web client as plain JS, red-lining every upstream-alerts suite test. Remove the parallel block, its page div and its six i18n keys; one state keeps one render surface, as the merge intended.
|
(ai: sglang-gpu0/qwen3.8-27b) CI 测试矩阵这一轮的 13 条红( 根因(人话):面板测试是把 web 客户端文件当纯 JS在沙箱里执行的,一行都不许带 TypeScript 类型标注。上一轮「丢弃本分支并行 banner、只留 master 已合入的 #5322 banner」的冲突解决,实际把两套 banner 都留下了——master 的 修法:按当初声明的意图把并行那套完整删掉(client.ts 渲染块 + page.ts 的 div + i18n 六个 key),报警状态恢复为单一渲染面(复用 #5322 的 master 继承的两条静态红(docs-sync / unused-exports)与本次无关,仍归 ci-fixes 支线(#5308/#5315/#5321)收敛。 |
(ai: vllm-qwen/qwen3.8-27b) 收到,知悉。这条是 tunnel DNS 那条 PR 的 CI 修复进展( |
… page div) f88fed9 deleted the client render block but left the six sys.dns_wedge_* catalog entries and the page div behind; tests/web-i18n.test.ts (#1024) rejects unreferenced catalog keys, which red-lined every CI shard carrying that test (ubuntu 2/3 windows 3/5). Remove the dead entries and the div.
|
(ai: sglang-gpu0/qwen3.8-27b) 上一轮(
补刀提交 |
|
(ai: sglang-gpu0/qwen3.8-27b)
也就是说:代码 × 文件集 × node 版本这三个乘积里找不到红。剩下的差异只有两件:Windows OS 本身(win-22-3 那片在 linux 上验不了真)和 CI runner 的环境(GITHUB_*/CI 变量、更弱的 CPU 导致的时序敏感)。日志 API 对这个 token 一律 404(绿 job 也一样),我拿不到失败的 求一条信息:失败 job 里具体是哪几个测试名红了(网页上展开任一个红 job 的 Step 输出,搜 |
📦 Merged Build PublishedThis merge to master was published as npm install -g billion-context@0.0.972-master.aaf4b1d # this exact build
npm install -g billion-context@master # always the newest merged build
|
Closes #5221 (and its duplicate #5222)
What & why
A long-lived proxy process can end up with a permanently wedged process-local name resolver after a WSL/VPN network switch: its libuv main threadpool's getaddrinfo state goes stale while the rest of the machine resolves fine. Every
/bili/tunnel admission then fails identically (unresolvable-> 502tunnel_destination_denied) until the PROCESS is restarted — restarting the host does not help in lanes that re-adopt the registered pid (launcher/dsh-web), which made this failure invisible and self-perpetuating (#5221/#5222, reporter-verified: kill pid -> instant recovery).Two layers, per the owner ruling of 2026-10-31 (no auto re-exec — "bili never kills itself" stays the product default):
Part 1 — admission stops false-denying. When the main pool exhausts all 3x2s attempts for a host, admission probes a FRESH worker-thread resolver (own threadpool) once, ~3s budget. Success proves the wedge is process-local; the worker's IPs flow through the EXACT same classification the main-pool answer would have (fe80 drop / self-port / linkLocal / private+allowlist) — the fallback admits nothing the main pool wouldn't admit. It also logs one loud line per episode telling the operator to restart the process: the probe mitigates ADMISSION only; outbound connects share the wedged pool (undici default connector), so a restart remains the full remedy.
workerFallback: falserestores the legacy fail-fast verdict byte-for-byte.Part 2 — detection + visibility (no behavior change). A persistent-wedge detector fed by the admission MAIN-pool outcomes only: only THROWING lookups count (an authoritative empty answer proves the resolver reachable); the gate is >=2 distinct hosts this process previously resolved OK, each now failing continuously for >=5 min. Tripped, it surfaces an alarm — web panel banner, additive
resolverWedgefield on/__bili/status+/__bili/overview, anacp_statusline — plus a warn log line re-logged every 10 min while the episode persists. The process itself keeps running exactly as before.Union of intent with PR #5322
PR #5322 implemented the same phase-2 scope in parallel (filed mid-flight against this issue family). This branch now carries BOTH phases as the single vehicle, adopting #5322's design wherever it is stronger: its module/wire shape (
{tripped, sinceMs, elapsedMs, hosts} | null, null = healthy), its stricter evidence rule (throw-only failures — my first cut miscounted authoritative empty answers), its ever-green gate (dropping my 24h recency window — more robust for multi-day processes), itsacp_statusalarm text, and its injected-clock test suite. What it lacked and this adds: a FIFO cap on the tracker (user-supplied/bili/<url>hosts are unbounded input; LRU-by-activity self-evicts under backdated clocks, verified by probe), hook placement BEFORE the worker stage (recording at verdict point would capture post-probe outcomes and miss exactly the wedges the probe rescues), the Part-1 worker probe itself, and the banner display cap. Recommend closing #5322 in favor of this PR — details in the comment there.Long-term "true non-restart self-heal" (fresh resolver covering admission AND the undici egress hot path) is tracked separately as design issue #5323 per the owner ruling — deliberately NOT bundled here.
Phase 1 predecessor: merged PR #5263 (error-message restart remedy + README section).
Behavior changes (disclosed per policy)
unresolvable502 message" and a fresh resolver thread also failed"clause when the probe ran; identical wording whenworkerFallback:false/__bili/status+/__bili/overviewresolverWedge:{tripped:true, sinceMs, elapsedMs, hosts} | null(null = healthy)acp_statusoutputConfig surface
None added. No new env var, config key, or CLI flag.
workerFallbackis a context-level seam (default ON) consumed internally; the detector has no user-facing knobs (constants live insrc/resolver-wedge.ts). The existing escape hatches (-F/BILI_UPSTREAM_PROXY) remain the documented remedies alongside restart.Tests
tests/resolver-wedge.test.ts— 7 tests, both gate directions (Test Blind Spot Retrospective: Reinforcing Negative Assertion Discipline, Tightening Fake Upstream Strictness, Thin Inst | 测试盲区复盘:补反向断言纪律、假上游严格化、薄安装冒烟、形态矩阵与生命周期覆盖 #3617): sustain-window boundary (4m59s vs 5m), single-host negative row, never-green hosts excluded, recovery below floor, streak anchoring, integration throughcheckTunnelDestination(throw-only evidence rule incl. the empty-answer case), FIFO cap bound.tests/tunnel-guard-worker-fallback.test.ts— 7 tests, both directions: wedge rescued (probe called exactly once), both-pools-fail keeps legacy verdict + extended wording, opt-out byte-identical legacy, classification parity rows (privateRemote / allowlist / linkLocal), real-worker plumbing (localhost resolves,.invalidrejects).workerFallback: falseto stay hermetic (no real DNS).Hot-file note: open PR #4983 also touches
src/tunnel-guard.ts(DNS caching, orthogonal layer); changes compose — rebase if it merges first.