Repository navigation
[PR] fix(observability): surface suspected DNS-resolver wedge as logged + panel alarm (#5221 phase 2) - #5322
Conversation
…WN_FINGERPRINT in LIFECYCLE.md (#4599) The lifecycle-ledger-guard job fails on every tree containing master since c8d7a80 (#4788 phase 2) and fc53fd4 (#4125): two BILI_* env vars read from src/ were never registered in LIFECYCLE.md, so every PR based on current master goes red within seconds (first observed on #5037). Backfill only, no code change: BILI_SESSION_CONTENT_STORE_PIN -> section 1 transition ledger (escape hatch for #4788 phase 2 idle-fold, default fold); BILI_SPAWN_FINGERPRINT -> section 2.2 internal spawn channel (#4125 spawner-side entry-script hash, launcher to child).
…n only) #5221 phase 2 per the owner ruling recorded in the #5222 thread: if it self-heals it does so WITHOUT a restart; a restart at most earns an alarm. So this lands detection + visibility and nothing else: - src/resolver-wedge.ts: per-process evidence state — a host qualifies only if this process saw it resolve OK before AND every attempt since has THROWN for >=5 min (an answered-empty lookup proves the resolver alive); >=2 such hosts in lockstep trip the alarm; recovery drops it on its own. - src/tunnel-guard.ts: feeds the detector from checkTunnelDestination (throwing resolutions = failure evidence, any answer = success). - surfaces, one helper feeding all of them: /__bili/status + /__bili/overview `resolverWedge` field, an instance-level line in acp_status beside the critical-advisory one, and a web banner beside the defect advisory (i18n en+zh). The log line carries the two manual workarounds (restart / -F upstream proxy). No config surface added; no process behavior change (availability stays exactly as today); kernel untouched. Tests: tests/resolver-wedge.test.ts with both gate directions asserted (#3617 discipline). Part of #5221; follow-up to #5263 (phase 1, merged).
a7892d6 to
2a8898e
Compare
📦 Built Package ArtifactBranch: Latest build — install this exact versionnpm install -g billion-context@0.0.7493-pr.5322.60242c6This exact version is immutable and reproducible. The rolling Recent builds (commit → version)
Download artifact
tar xzf billion-context-pr5322.tgz
npm install -g packageInstall instructions are refreshed on each push. |
(ai: vllm-qwen/qwen3.8-27b) 查重: 不重复。 本 PR 是 #5221 phase 2 的唯一 PR。相关条目: phase 1 的 #5263 已于今日 07:19 UTC 合入 master(issue 正文里 "still open" 是过时信息);兄弟 PR #5272(Closes #5222,worker-thread fresh-resolver probe)机制不同(自愈 vs 报警)、目标 issue 不同,按政策不并入——但两者有语义耦合,见下方警告。 标签: 八问
|
…s & surfaces (union of intent, #5221) PR #5322 (filed against this same issue family while this branch was in flight) landed a parallel phase-2 implementation with stricter evidence rules than my first cut. Per union-of-intent, adopt its design where it is stronger and keep what it lacks: Adopted from #5322: - module design & wire shape: getResolverWedgeState() -> {tripped, sinceMs, elapsedMs, hosts} | null (null = healthy) instead of my always-present object; injected-clock test seam (_resolverWedgeTest) - evidence rule: only THROWING lookups count as failures — an authoritative empty answer proves the resolver reachable (#1686 boundary), which my "empty = failure" version would have miscounted - gate: host must have resolved OK at least once in-process (no recency window — more robust for multi-day processes than a 24h cutoff) - acp_status alarm line (their text verbatim) - their full test suite (6 tests), plus workerFallback:false pin on the integration test to stay hermetic next to the default-on worker probe Kept / added on top: - FIFO cap (MAX_TRACKED_HOSTS=256): user-supplied /bili/<url> hosts are an unbounded input; LRU-by-activity self-evicts under backdated clocks (verified by probe), first-seen FIFO does not - hook placement BEFORE the worker stage: recording at verdict point (as #5224's message work implies) would capture post-probe outcomes and miss exactly the wedges the probe rescues Tests: tests/resolver-wedge.test.ts now 7/7 (theirs 6 + cap bound); tests/tunnel-guard*.test.ts 28/28; typecheck clean; build green. #5221 (phase 2 per owner ruling 2026-10-31); Closes #5221 (and its duplicate #5222)
CI review of this PR: the unused-exports gate flags ResolverWedgeWire as INTERNAL_ONLY — the wire-shape type is declared and consumed only inside src/resolver-wedge.ts (getResolverWedgeState's return type); no other module imports it by name. Drop the leading export per the gate's own prescription; runtime behavior unchanged (type-only).
(ai: vllm-qwen/qwen3.8-27b) CI 红分账完成(首轮 3 条车道失败,逐条核实并处置,新 head
审核结论不变:可合并(等 CI 绿)。测试矩阵在本轮因静态门禁红被 fail-fast 跳过,重触发后会真正跑起来——若分片出现与在档 born-red 集群同签名的失败,按 #4599 既有清单对号入座,与本 PR diff 无关。 |
📦 Merged Build PublishedThis merge to master was published as npm install -g billion-context@0.0.957-master.485c5ea # this exact build
npm install -g billion-context@master # always the newest merged build
|
(ai: vllm-qwen/qwen3.8-27b) Convergence notice: this PR's phase-2 scope has been folded into PR #5272 (union of intent) — recommend closing this PR as superseded. PR #5272 carried the other half of the same issue family (the admission-side worker probe) while this PR implemented the detection+visibility scope in parallel. Rather than let two PRs fight over Adopted from this PR:
Gap fixed on top (in #5272):
Per the loop guard (one mechanism family = one PR), #5272 is now the single vehicle for both phases. Recommend closing this PR as superseded — closing is owner-only per policy. No further commits will land here. |
… of intent, #5222/#5326) - web surfaces (client/i18n/page/admin): take master's merged #5322 detector surface; drop this branch's parallel dns-wedge-banner implementation — one alarm state, one rendered surface. - resolver-wedge.ts + test: keep this branch's superset (FIFO cap MAX_TRACKED_HOSTS=256 against user-supplied host growth). - tunnel-guard.ts: union — main-pool verdict recorded BEFORE the worker stage (a worker-rescued admission still counts as wedge evidence, #5326 coupling) + immediate getResolverWedgeState() eval on failure preserved from #5322 (warn log fires at trip moment, not next status poll). - tests/resolver-wedge.test.ts: workerFallback:false on the checkTunnelDestination scenario keeps it hermetic with the worker stage in.
…er merge The union-of-intent merge kept BOTH alarm render surfaces for the same resolver-wedge state: master's merged #5322 banner (resolver-wedge-banner) and this branch's parallel implementation (dns-wedge-banner). The parallel block carried a TS parameter annotation ((h: string)) which breaks the panel harness that evaluates the web client as plain JS, red-lining every upstream-alerts suite test. Remove the parallel block, its page div and its six i18n keys; one state keeps one render surface, as the merge intended.
Part of #5221 (phase 2; phase 1 is PR #5263 on the same branch lineage, merged). Follow-up per owner ruling (comment-6540576478): automatic self-restart is OUT of scope — detector + alarm surface ship instead.
What
When this host resolver has been answered-then-broken (WSL/VPN class, #5221), the proxy now:
src/resolver-wedge.ts, new): >=2 distinct hosts that resolved successfully in THIS process are all failing every subsequent lookup for >=5 min sustained → suspected DNS wedge. An authoritative EMPTY answer proves the resolver alive and does NOT count (the empty-answer branch of tunnel-guard was mislabeled as failure before).[resolver-wedge] suspected local-resolver wedge: N host(s) this process resolved before now fail every lookup for >=5 min (...) — model requests 502 until the proxy process restarts or egress routes through an upstream proxy (-F/--forward-proxy); see #5221/__bili/status+/__bili/overviewfieldresolverWedge: {tripped, hosts, sinceMs, elapsedMs}(nullwhile not tripped) + web panel banner next to the update-advisory banner, same style.No auto-restart, no config knob, no wire behavior change. Remedy for a wedged instance remains a restart — this PR makes the wedge VISIBLE where people actually look, with cause + remedy inline.
Files
src/resolver-wedge.ts(new): in-memory detector; per-host last-outcome + first-failure streak; time injectable for tests.src/tunnel-guard.ts: success/failure bookkeeping feeds the detector (throw = failure, any answer incl. empty = alive); tunnel verdict shape untouched.src/admin.ts: status/overview fields.src/web/client.ts(+ i18n en/zh): banner row.tests/resolver-wedge.test.ts(new): 6 rows, gate both directions (Test Blind Spot Retrospective: Reinforcing Negative Assertion Discipline, Tightening Fake Upstream Strictness, Thin Inst | 测试盲区复盘:补反向断言纪律、假上游严格化、薄安装冒烟、形态矩阵与生命周期覆盖 #3617): sustained multi-host trips / single-host never / never-green hosts never / recovery drops below floor / streak anchoring / e2e through checkTunnelDestination.Local: typecheck clean; targeted suites green (resolver-wedge 6 + tunnel-guard 17 + acp-status + advisory-parity = 37). Full verdict = CI on pushed head. Kernel untouched. No config-surface change.
Known residual (tracked separately)
Detection rides the TUNNEL lane only (#2754 default): direct-egress requests bypassing the guard feed nothing yet. The true non-restart heal (fresh-resolver covering admission AND egress) is filed as its own design issue.