Skip to content

[PR] fix(observability): surface suspected DNS-resolver wedge as logged + panel alarm (#5221 phase 2) - #5322

Merged
ranxianglei merged 4 commits into
masterfrom
2026-10-31_dns-wedge-alarm
Oct 31, 2026
Merged

ranxianglei merged 4 commits into
masterfrom
2026-10-31_dns-wedge-alarm

Conversation

@ranxianglei

@ranxianglei ranxianglei commented Oct 31, 2026 •

Copy link
Copy Markdown
Owner

Part of #5221 (phase 2; phase 1 is PR #5263 on the same branch lineage, merged). Follow-up per owner ruling (comment-6540576478): automatic self-restart is OUT of scope — detector + alarm surface ship instead.

What

When this host resolver has been answered-then-broken (WSL/VPN class, #5221), the proxy now:

  1. detects it (src/resolver-wedge.ts, new): >=2 distinct hosts that resolved successfully in THIS process are all failing every subsequent lookup for >=5 min sustained → suspected DNS wedge. An authoritative EMPTY answer proves the resolver alive and does NOT count (the empty-answer branch of tunnel-guard was mislabeled as failure before).
  2. logs at WARN (once per episode, re-logged at most every 10 min while wedged): [resolver-wedge] suspected local-resolver wedge: N host(s) this process resolved before now fail every lookup for >=5 min (...) — model requests 502 until the proxy process restarts or egress routes through an upstream proxy (-F/--forward-proxy); see #5221
  3. surfaces it in every alarm channel: /__bili/status + /__bili/overview field resolverWedge: {tripped, hosts, sinceMs, elapsedMs} (null while not tripped) + web panel banner next to the update-advisory banner, same style.

No auto-restart, no config knob, no wire behavior change. Remedy for a wedged instance remains a restart — this PR makes the wedge VISIBLE where people actually look, with cause + remedy inline.

Files

Local: typecheck clean; targeted suites green (resolver-wedge 6 + tunnel-guard 17 + acp-status + advisory-parity = 37). Full verdict = CI on pushed head. Kernel untouched. No config-surface change.

Known residual (tracked separately)

Detection rides the TUNNEL lane only (#2754 default): direct-egress requests bypassing the guard feed nothing yet. The true non-restart heal (fresh-resolver covering admission AND egress) is filed as its own design issue.

@ranxianglei ranxianglei changed the title fix(observability): surface suspected DNS-resolver wedge as logged + panel alarm (#5221 phase 2) [PR] fix(observability): surface suspected DNS-resolver wedge as logged + panel alarm (#5221 phase 2) Oct 31, 2026
ework-agent and others added 2 commits October 31, 2026 16:16
…WN_FINGERPRINT in LIFECYCLE.md (#4599)

The lifecycle-ledger-guard job fails on every tree containing master since c8d7a80 (#4788 phase 2) and fc53fd4 (#4125): two BILI_* env vars read from src/ were never registered in LIFECYCLE.md, so every PR based on current master goes red within seconds (first observed on #5037).

Backfill only, no code change: BILI_SESSION_CONTENT_STORE_PIN -> section 1 transition ledger (escape hatch for #4788 phase 2 idle-fold, default fold); BILI_SPAWN_FINGERPRINT -> section 2.2 internal spawn channel (#4125 spawner-side entry-script hash, launcher to child).
…n only)

#5221 phase 2 per the owner ruling recorded in the #5222 thread: if it
self-heals it does so WITHOUT a restart; a restart at most earns an alarm.
So this lands detection + visibility and nothing else:

- src/resolver-wedge.ts: per-process evidence state — a host qualifies only
  if this process saw it resolve OK before AND every attempt since has
  THROWN for >=5 min (an answered-empty lookup proves the resolver alive);
  >=2 such hosts in lockstep trip the alarm; recovery drops it on its own.
- src/tunnel-guard.ts: feeds the detector from checkTunnelDestination
  (throwing resolutions = failure evidence, any answer = success).
- surfaces, one helper feeding all of them: /__bili/status + /__bili/overview
  `resolverWedge` field, an instance-level line in acp_status beside the
  critical-advisory one, and a web banner beside the defect advisory
  (i18n en+zh). The log line carries the two manual workarounds (restart /
  -F upstream proxy).

No config surface added; no process behavior change (availability stays
exactly as today); kernel untouched. Tests: tests/resolver-wedge.test.ts
with both gate directions asserted (#3617 discipline).

Part of #5221; follow-up to #5263 (phase 1, merged).
@ranxianglei
ranxianglei force-pushed the 2026-10-31_dns-wedge-alarm branch from a7892d6 to 2a8898e Compare October 31, 2026 08:25
@github-actions

github-actions Bot commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-31_dns-wedge-alarm (1cc57ed)

Latest build — install this exact version

npm install -g billion-context@0.0.7493-pr.5322.60242c6

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Recent builds (commit → version)

commit version build
1cc57ed billion-context@0.0.7493-pr.5322.60242c6 build
2a8898e billion-context@0.0.7463-pr.5322.5c1c5c7 build

Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr5322.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重: 不重复。 本 PR 是 #5221 phase 2 的唯一 PR。相关条目: phase 1 的 #5263 已于今日 07:19 UTC 合入 master(issue 正文里 "still open" 是过时信息);兄弟 PR #5272(Closes #5222,worker-thread fresh-resolver probe)机制不同(自愈 vs 报警)、目标 issue 不同,按政策不并入——但两者有语义耦合,见下方警告。

标签: bug + scope/wiring + p3(缺陷谱系的观测性修复;机制在请求管道/tunnel admission 路径;本 PR 本身不恢复可用性——那是 #5221/#5272 的 p1,此条目只是「让 wedge 可见」的低频打磨)。

八问

  1. 是什么问题: WSL/VPN 变更后长驻代理的进程内 DNS resolver 永久失效(Long-lived proxy wedges on local DNS after WSL network/VPN changes — tunnel admission stays unresolvable (502) until process restart #5221),模型流量全 502,而系统里没有任何地方说明发生了什么、该怎么办——用户只能盲猜重启。本 PR 让这个状态可见。
  2. 什么原因导致的: resolver 状态陈旧本身是 OS/环境层现象(根因归 Long-lived proxy wedges on local DNS after WSL network/VPN changes — tunnel admission stays unresolvable (502) until process restart #5221);这里缺的是检测与上报层——tunnel-guard 只逐次判 admission,从不记录「这个 host 此前解析成功过」,所以「先成功后失败」的签名不可观测。
  3. 影响范围与严重程度: 仅影响真正发生 wedge 的实例(WSL/VPN 类,低频);严重程度为体验级——底层不可用性仍是 Long-lived proxy wedges on local DNS after WSL network/VPN changes — tunnel admission stays unresolvable (502) until process restart #5221 下的 p1,本 PR 只增加可见性,不改任何行为。
  4. 怎么修复的: 内存检测器(src/resolver-wedge.ts):per-host 记录最后成功时间 + 首次失败锚点;≥2 个曾成功解析的 host 连续 ≥5 min lookup 失败 → 判定疑似 wedge;权威空答案计为「resolver 活着」不计入(顺带纠正了 tunnel-guard 空答分支旧有的误标)。三个报警面: WARN 日志(每 episode 一次、持续卡住每 10 min 重发)、/__bili/status + /__bili/overview 字段 resolverWedge、web panel banner(与 update-advisory banner 同风格)。无自动重启、无配置项、wire 行为不变。
  5. 是否有回归问题: 逐项审了触碰路径的既有行为——checkTunnelDestination 两分支返回值与改动前逐字节一致(detector 纯旁路记账);两个 JSON 端点仅新增字段(未触发时 null);acp_status 行与 panel banner 仅在触发时出现;kernel/config/version/lockfile 未动。advisory-endpoint-parity 本地通过(该套件断言 overview/status 字段对等,两侧同加不破对等)。hosts Map 单调增长但受 distinct upstream host 数约束(有限小集合),判断可接受。高危模式逐项过:无语义漂移、无 fail-fast 变吞错、无阈值/时序/输出格式变化。
  6. 是否破坏缓存: 不涉及——prefix cache、重试、去重策略均不受影响。
  7. 是否改了设计方向: 不涉及——纯观测性附加,无架构转向。
  8. 是否引入新配置: 无。

⚠️ 与兄弟 PR #5272 的语义耦合(需 owner 注意)

#5272 给同一函数加「fresh worker-thread resolver probe」——worker 成功则 admission 放行(进程内 wedge 自愈)。#5272 落地后,被救的请求会走正常路径 → 本 PR 的检测器收到「成功」信号 → 在这个报警要覆盖的进程内 wedge 场景里,报警会静默(此时出站模型流量仍坏、仍需重启);反过来机器级 DNS 故障时 worker 也失败,本 PR 报警触发,但它给出的「重启可修」remedy 在该场景不准确。两个 PR 单独各自自洽,组合后互相误导。建议: 本 PR 先行合并(面小、CI 已在跑),#5272 随后 rebase(它当前 mergeable_state=DIRTY,本来就要 rebase)并调整 hook 语义——主池耗尽即使被 worker 救起也应记为失败证据,理想情况直接消费 probe 结果做更准的诊断。这不影响本 PR 单独的正确性。

PR body 事实修正(披露)

原 body 三处与代码不符,已直接 PATCH 更正(未动代码): ①字段名写作 suspectedResolverWedge,代码实际是 resolverWedge;②日志引用与实际文案不一致(已换成真实文案);③"logs once",实际是持续 wedge 期间每 10 min 重发。另将 "phase 1 … still open" 更新为 merged(已核实)。

验证

结论

方向符合 owner 裁决(检测+报警,不做自动重启);diff 干净单 commit,测试纪律达标(#3617 双向门禁、边界精确到秒、e2e 走真函数);无回归、无配置面变更。可合并(等 CI 绿);建议顺序: 本 PR 先合,#5272 之后 rebase 并按上述调整 hook 语义。

@ranxianglei ranxianglei added bug Something isn't working p3 scope/wiring labels Oct 31, 2026
ranxianglei pushed a commit that referenced this pull request Oct 31, 2026
…s & surfaces (union of intent, #5221)

PR #5322 (filed against this same issue family while this branch was in
flight) landed a parallel phase-2 implementation with stricter evidence
rules than my first cut. Per union-of-intent, adopt its design where it is
stronger and keep what it lacks:

Adopted from #5322:
- module design & wire shape: getResolverWedgeState() -> {tripped, sinceMs,
  elapsedMs, hosts} | null (null = healthy) instead of my always-present
  object; injected-clock test seam (_resolverWedgeTest)
- evidence rule: only THROWING lookups count as failures — an authoritative
  empty answer proves the resolver reachable (#1686 boundary), which my
  "empty = failure" version would have miscounted
- gate: host must have resolved OK at least once in-process (no recency
  window — more robust for multi-day processes than a 24h cutoff)
- acp_status alarm line (their text verbatim)
- their full test suite (6 tests), plus workerFallback:false pin on the
  integration test to stay hermetic next to the default-on worker probe

Kept / added on top:
- FIFO cap (MAX_TRACKED_HOSTS=256): user-supplied /bili/<url> hosts are an
  unbounded input; LRU-by-activity self-evicts under backdated clocks
  (verified by probe), first-seen FIFO does not
- hook placement BEFORE the worker stage: recording at verdict point (as
  #5224's message work implies) would capture post-probe outcomes and miss
  exactly the wedges the probe rescues

Tests: tests/resolver-wedge.test.ts now 7/7 (theirs 6 + cap bound);
tests/tunnel-guard*.test.ts 28/28; typecheck clean; build green.

#5221 (phase 2 per owner ruling 2026-10-31); Closes #5221 (and its duplicate #5222)
ci-check added 2 commits October 31, 2026 17:41
CI review of this PR: the unused-exports gate flags ResolverWedgeWire as
INTERNAL_ONLY — the wire-shape type is declared and consumed only inside
src/resolver-wedge.ts (getResolverWedgeState's return type); no other module
imports it by name. Drop the leading export per the gate's own prescription;
runtime behavior unchanged (type-only).
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

CI 红分账完成(首轮 3 条车道失败,逐条核实并处置,新 head 1cc57ed9d 已推送):

  1. unused-exports — 本 PR 自带缺陷,已修:新文件里 ResolverWedgeWire 类型带 export 但全仓只有模块内部引用,门禁判 INTERNAL_ONLY。已按门禁处方去掉 export(独立标注 commit e212ccd9d,type-only,行为零变化);
  2. lifecycle-ledger-guard — 继承 master 红,已解:缺账的 BILI_SESSION_CONTENT_STORE_PIN / BILI_SPAWN_FINGERPRINT 与 master 自身红同签名(CI Control Center: Centralized CI Failure Handling & Status Sync (Comment all CI issues here; do not open new issues) | CI 总控: CI 失败集中处理与状态同步(所有 CI 问题在此评论, 勿开新 issue) #4599 floor 710/711),按 CI 协调政策 merge ci-fixes@0371089db 进分支,本地复跑 OK;
  3. docs-sync + unused-exports(codex-models) — 继承 master 红,待哨兵收敛:前者是 New built-in chat prompt pack: default compression rules for long-term companionship / multi-session memory scenarios (e | 新增内置 chat 提示压缩包:长期陪伴 / 多会话记忆场景的默认压缩规则(compress.promptPack="chat" 启用) #5319(chat-pack)改了文档没同步种子 yaml 的 born-red(实测 merge ci-fixes@0371089db 后仍红,该修复尚未进 ci-fixes),后者是 CodexWindowSource 僵尸导出(CI Control Center: Centralized CI Failure Handling & Status Sync (Comment all CI issues here; do not open new issues) | CI 总控: CI 失败集中处理与状态同步(所有 CI 问题在此评论, 勿开新 issue) #4599 floor 714 已报)。两条均已带完整签名报到 CI Control Center: Centralized CI Failure Handling & Status Sync (Comment all CI issues here; do not open new issues) | CI 总控: CI 失败集中处理与状态同步(所有 CI 问题在此评论, 勿开新 issue) #4599,等哨兵收敛后本分支再 merge 一次即全绿。

审核结论不变:可合并(等 CI 绿)。测试矩阵在本轮因静态门禁红被 fail-fast 跳过,重触发后会真正跑起来——若分片出现与在档 born-red 集群同签名的失败,按 #4599 既有清单对号入座,与本 PR diff 无关。

@ranxianglei
ranxianglei merged commit 485c5ea into master Oct 31, 2026
38 of 40 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

📦 Merged Build Published

This merge to master was published as 0.0.957-master.485c5ea (rolling master tag).

npm install -g billion-context@0.0.957-master.485c5ea  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Convergence notice: this PR's phase-2 scope has been folded into PR #5272 (union of intent) — recommend closing this PR as superseded.

PR #5272 carried the other half of the same issue family (the admission-side worker probe) while this PR implemented the detection+visibility scope in parallel. Rather than let two PRs fight over src/tunnel-guard.ts / src/resolver-wedge.ts, the union was taken INTO #5272, adopting this PR's design wherever it is stronger:

Adopted from this PR:

  • module & wire shape: getResolverWedgeState() → {tripped, sinceMs, elapsedMs, hosts} | null (null = healthy)
  • evidence rule: only THROWING lookups count as failures — an authoritative empty answer proves the resolver reachable (a first cut elsewhere miscounted those as wedge evidence)
  • gate: host must have resolved OK at least once in-process (no recency window — more robust for multi-day processes than a 24h cutoff)
  • the acp_status alarm wording (verbatim)
  • the injected-clock test seam (_resolverWedgeTest().setNow) and its full 6-test suite

Gap fixed on top (in #5272):

  • unbounded tracker: user-supplied /bili/<url> hosts are unbounded input → FIFO cap of 256 entries (LRU-by-activity self-evicts under backdated clocks, verified by probe)
  • hook placement moved BEFORE the worker stage: recording at the verdict point would capture post-probe outcomes and miss exactly the wedges the probe rescues
  • plus the part-1 worker probe itself and the panel display cap, which this PR does not cover

Per the loop guard (one mechanism family = one PR), #5272 is now the single vehicle for both phases. Recommend closing this PR as superseded — closing is owner-only per policy. No further commits will land here.

ranxianglei pushed a commit that referenced this pull request Oct 31, 2026
… of intent, #5222/#5326)

- web surfaces (client/i18n/page/admin): take master's merged #5322 detector
  surface; drop this branch's parallel dns-wedge-banner implementation — one
  alarm state, one rendered surface.
- resolver-wedge.ts + test: keep this branch's superset (FIFO cap
  MAX_TRACKED_HOSTS=256 against user-supplied host growth).
- tunnel-guard.ts: union — main-pool verdict recorded BEFORE the worker stage
  (a worker-rescued admission still counts as wedge evidence, #5326 coupling)
  + immediate getResolverWedgeState() eval on failure preserved from #5322
  (warn log fires at trip moment, not next status poll).
- tests/resolver-wedge.test.ts: workerFallback:false on the
  checkTunnelDestination scenario keeps it hermetic with the worker stage in.
ranxianglei pushed a commit that referenced this pull request Oct 31, 2026
…er merge

The union-of-intent merge kept BOTH alarm render surfaces for the same
resolver-wedge state: master's merged #5322 banner (resolver-wedge-banner)
and this branch's parallel implementation (dns-wedge-banner). The parallel
block carried a TS parameter annotation ((h: string)) which breaks the
panel harness that evaluates the web client as plain JS, red-lining every
upstream-alerts suite test. Remove the parallel block, its page div and its
six i18n keys; one state keeps one render surface, as the merge intended.
@github-actions github-actions Bot mentioned this pull request Oct 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working p3 scope/wiring

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants