Skip to content

[PR] fix: refresh skipped-proxy evidence in repeated alerts - #7370

Merged
ranxianglei merged 1 commit into
ranxianglei:masterfrom
Afloat16:2026-11-20_refresh-skipped-proxy-alerts
Nov 20, 2026
Merged

ranxianglei merged 1 commit into
ranxianglei:masterfrom
Afloat16:2026-11-20_refresh-skipped-proxy-alerts

Conversation

@Afloat16

Copy link
Copy Markdown
Contributor

Problem

An active upstream alert could keep its old skipped-proxy endpoint after later requests started using that proxy or deliberately bypassed it. Its English hint was refreshed, but the localized web banner continued reading the obsolete structured fields and giving incorrect recovery advice.

Changes

Replace the optional skipped-proxy fields with the latest failure's evidence, including clearing them when absent. The alert's instance, count, timestamps and failure code stay intact. A later skipped proxy still replaces the previous endpoint/source with the existing credential redaction.

This is one alert-state correction shared by the JSON overview and the localized banner. Routing, retry and circuit policies are unchanged.

Verification

  • Tested base: 1e357bf2d6632bfbd2a74ffb70ecee1af1ec7cc3 (current master, 0.1.216).
  • Before the fix: 4 behavior assertions fail, 26 controls pass. Three failures use actual proxy decisions for explicit direct, auto and host bypass; the fourth runs the real web-client IIFE against the HTTP overview endpoint and observes obsolete advice.
  • After the fix: 71 targeted tests pass across skipped-proxy hints, upstream routing, alerts, the health endpoint, circuit handling and failure egress. A changing-proxy control confirms the new endpoint is still shown and credentials remain redacted.
  • npm run typecheck, npm run build, npm run check:unused-exports, npm run check:test-import-graph and git diff --check: passed.
  • Published tree ba9f0bff05ff781150fc40964fadb970b0e5a781 matches the locally validated tree.
  • Node 24.22.3 / Linux x86_64. No live host-client or Windows run; full regression evidence remains repository CI on this head.

Related #2860 and #5574. The former introduced the structured evidence, while the latter expires inactive rows; this corrects the evidence of an alert that is still actively failing.

Fixes #7369

@ranxianglei ranxianglei changed the title fix: refresh skipped-proxy evidence in repeated alerts [PR] fix: refresh skipped-proxy evidence in repeated alerts Nov 20, 2026
@ranxianglei ranxianglei added root-cause bug Something isn't working p3 scope/ui labels Nov 20, 2026
@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重: 不重复. Searched open+closed issues/PRs (GitHub + mirror) for the same "stale skipped-proxy evidence" symptom. The only open PR addressing #7369 is this one (#7370). Related closed work is distinct, not parallel: #2860 (introduced the structured skippedProxy/skippedSource fields), #2809 (original diagnosis), #5574 (5-min TTL expiry for inactive rows — this PR covers a row that is still actively failing). No other PR/issue fixes this same symptom+mechanism.

标签: added bug p3 scope/ui (kept the existing root-cause). p3 because it is a diagnostic/advisory banner surface giving wrong recovery advice in an edge scenario (route changes mid-outage) — the model traffic path, routing, retry and circuit policies are untouched. Judgment call; bump to p2 if you read the misleading advice as a real functional defect.

Local verification (independent of CI):

  • Reverted the 2-line fix locally → exactly 4 behavior assertions fail, 26 controls pass (matches your claim). Restored → all 30 pass in the two touched files.
  • Full targeted suite across the 6 alert/routing/egress test files: 71 pass, 0 fail.
  • CI on this head (7f0bd0e): still queued (22 queued / 8 skipped) at my last check — the full-regression green is CI's to confirm.

8问 (structured review):

  1. 是什么问题: 上游告警横幅(web 面板)在请求路由变化后(例如从「默认直连但环境有代理」切到 auto / 显式 direct / NO_PROXY 命中)仍保留旧的 skipped-proxy 恢复建议——英文 hint 刷新了,但本地化横幅读的是结构化字段,继续推荐一个已在用或被刻意绕过的代理。
  2. 什么原因导致的: 根因在 recordUpstreamAlert 的更新路径(src/upstream-alerts.ts:119-126)。同一 (kind, host) 的重复失败若没有 skipped 信息,skippedFields 为 {},Object.assign(existing, {}) 是空操作——早先写入的 skippedProxy/skippedSource 不会被清除;而英文 hint 在 :122 被刷新。两者从此分叉:hint 反映最新失败,结构化字段(横幅真正读的东西)停留在第一次设置时的值。
  3. 影响范围与严重程度: 仅 web 横幅的本地化恢复提示 + JSON overview 的 skippedProxy/skippedSource 字段。触发条件:同一 host+失败类型持续失败,且两次失败之间代理决策发生变化。严重程度 p3——诊断/建议面在边缘场景给出错误建议;模型流量路径、路由、重试、熔断策略完全不受影响,不是可用性缺陷。
  4. 怎么修复的: 在根因处(告警更新路径)先清旧再写新:delete existing.skippedProxy; delete existing.skippedSource; 再 Object.assign(existing, skippedFields)。使告警始终反映最新一次失败的路由证据——这正是 [Feature] Web UI 主动提示上游连接失败(全局 banner),而不是只埋日志 #1682「告警行 = 当前时刻快照」语义的自洽化。后续若某次失败确实跳过了另一个代理,仍会用既有脱敏替换旧端点/来源(测试已钉住)。
  5. 是否有回归: 逐项过触碰路径的既有行为——新建告警行(:138)不变;重复失败带 skipped 时 delete+assign = 净 assign(新旧一致,新覆盖旧)不变;code/count/firstSeen/lastSeen/hint 处理不变。高危模式排查:无 fail-fast 变吞错、无阈值/时序漂移。唯一行为变化即本 PR 目标(陈旧字段被清除),无意外回归。
  6. 是否破坏缓存: 不涉及——未触碰任何缓存/重试/增量游标/去重键;告警表是内存态、无持久化。
  7. 是否改了设计方向: 没有。强化既有「快照即当前」设计,让结构化证据与 hint 一致;无架构/边界/数据流转向。
  8. 是否引入了新的配置: 没有——无新 env/config/CLI 面。

行为变更披露(旧 → 新): 旧——无 skipped 的重复失败保留早先 skippedProxy/skippedSource(仅英文 hint 刷新),本地化横幅持续给出陈旧建议;新——告警的 skippedProxy/skippedSource 始终反映最新失败:最新未跳过代理则字段被清除(JSON 中缺失、横幅无本地化注记),最新跳过了代理则用脱敏值替换。原因:#7369 要求结构化证据与 hint 一致。

测试有效性: 4 条新行为断言(3 个路由切换走生产 resolveProxyDecision + 1 个真实 web-client IIFE 打 HTTP overview 端点)+ 1 条「后续代理替换旧脱敏端点」控制项。断言有牙:本地回退修复即精确 4 红、26 控制绿,非恒真。

diff 整洁度 / 代码整洁: 3 文件、83 增 0 删,无跑题改动、无空行/重排噪音;命名达意、无死代码、无魔法数、无新依赖、风格与仓库一致。外部提交质量达标,无需对齐重构。

判定: 可合(mergeable),待 CI 全绿确认。 修复正确、最小、打在根因层,回归覆盖有真实断言支撑;路由/重试/熔断零改动。

@ranxianglei
ranxianglei merged commit 5aaefd5 into ranxianglei:master Nov 20, 2026
120 of 156 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working p3 root-cause scope/ui

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Upstream banner retains skipped-proxy advice after the request route changes

2 participants