Repository navigation
[PR] fix: expire stale upstream alert rows after 5 min (DSH overlay stuck on dead loopback port) - #5574
Conversation
…on dead loopback port) An upstream-failure alert row was removed only by a successful fetch to the same host or a process restart (#1682). A single stray ECONNREFUSED against a loopback provider shim that rebound to a fresh ephemeral port after a plugin reload therefore pinned the "upstream unavailable" banner forever: the dead port can never succeed again, so the success-clear path could never fire for it (evidence: count:1, circuit:null, firstSeen==lastSeen). Add a staleness bound: rows whose LAST failure predates now - ALERT_STALE_MS (5 min) are pruned lazily inside both read projections (the /__bili/upstreams health report that feeds the DSH overlay, and the /__bili/overview alerts list that feeds the web banner). Hosts still failing refresh lastSeen on every failure (live outages; open-circuit half-open probes re-record through the relay path every 60s), so they never age out — only true zombies die. No config surface added; the bound is uniform rather than loopback-only because freshness keying is on-model regardless of IP class. Closes #5572
📦 Built Package ArtifactBranch: Latest build — install this exact versionnpm install -g billion-context@0.0.8028-pr.5574.d01b5bbThis exact version is immutable and reproducible. The rolling Recent builds (commit → version)
Download artifact
tar xzf billion-context-pr5574.tgz
npm install -g packageInstall instructions are refreshed on each push. |
(ai: vllm-qwen/qwen3.8-27b) 查重:无 —— 检索了 open+closed 的 issue 与 PR(stale upstream alert / overlay banner stuck / ECONNREFUSED loopback 等关键词组合),没有同症状或同机制的既有条目;本 PR 就是 #5572 的修复本体。 标签:bug + dsh + scope/ui + p2 —— 一次性失败把「上游不可用」横幅钉死到重启,属个别功能异常且有重启绕路,非流量/数据问题。 这是本舰队的 agent PR( |
Dead import, pre-existing on master: server.ts never references getUpstreamAlerts — both banner reads live in src/server/admin.ts (overview + /__bili/upstreams). Found while reviewing #5574, which changes this function's signature; removing it keeps the changed surface clean. One line, no behavior change.
|
(ai: vllm-qwen/qwen3.8-27b) APPROVE —— 独立审完:diff 全量通读、两个横幅消费链路端到端追源、relay 记录/清除路径逐一核对、定向测试本地重跑。结论按八问。 审核过程(独立产出)
既有行为回归清点(逐项结论)
有意行为变更(PR 已披露,我核过表述准确):旧=告警行直到「对该 host 成功 fetch」或进程重启才消失(死端口≈永久);新=最后失败 >5min 的行从两个横幅消失,下次失败再出现。持续宕机 host 不受影响(失败不断刷新 lastSeen)。边界情形(客户端停重试 >5min → 行消失)符合「当前活跃快照」语义,披露到位。 八问
已顺手修: 判定:可合并(merge 由人执行)。②③ 两个范围外方向维持 issue 侧 🔶 待 owner 拍板,不影响本 PR。 |
(ai: vllm-qwen/qwen3.8-27b) review 完成:APPROVE,可合并(merge 由人执行)。 独立审完 diff + 两个横幅消费链路 + relay 记录/清除路径;CI 绿(镜像缓存 head 8b05d23,failed 0/pending 0);本地定向 5 套件 47/47 + typecheck 干净;3 个新回归断言有牙且双向。已顺手修: |
(ai: vllm-qwen/qwen3.8-27b) review 车道收工:上面这条即本轮独立审核的最终结论(从 PR 评论镜像回来),无新增发现、无待办。PR 待 owner 合并;②③ 两个范围外方向在 issue 侧 🔶 等你拍板,不影响本 PR。 |
(ai: vllm-qwen/qwen3.8-27b) PR #5574 审核结论(Closes #5572 · head
行为变更披露(旧→新):旧——告警行直到「对该 host 成功 fetch」或进程重启才消失(死端口 ≈ 永久);新——最后一次失败 >5min 的行从两个横幅消失。原因:让表的实际寿命对齐其声明的「当前时刻活跃快照」语义,修复横幅永久卡死。持续宕机的公共上游不受影响(失败不断刷新 lastSeen)。 验证:
判定:代码层面审核通过——根因定位准确、修法最小且语义自洽、回归清点干净、行为变更完整披露。可合并,待人工批准(GitHub 禁止自批;另 §7.4:新增默认时序上界属产品决策,人工门禁)。继承红是否等哨兵收敛后再合由 owner 定(先例 #4441:同类小 delta merge ci-fixes 后全绿)。 |
…-driven, not a 60s timer The #5572 fix comment said open-circuit half-open probes "re-record every 60s" to keep live-outage rows fresh. That is wrong: upstreamCircuitBlockReason is consulted per-request and there is no background timer in the module, so a host no session references anymore (provider deselected / removed / model switched away) gets neither fresh failures nor probes, and lastSeen freezes. Field report on #5572 confirmed it: circuit open, nextProbeInMs 0, lastSeen unchanged for 25+ min. Corrected the comment to state the real invariant — ongoing failing requests refresh lastSeen; an abandoned host ages out via the TTL, which is its only exit. Comment-only; no behavior change.
|
(ai: vllm-qwen/qwen3.8-27b) APPROVE(第 2 轮,head aedb2bc) —— 自上一轮批准以来分支新增内容只有两类:① master 合入( 增量审核(独立产出)
八问(相对第 1 轮的答案均保持有效,此处给增量判定)
一个非阻塞观察(不在本 PR 处置范围,仅记录): 判定:可合并(merge 由人执行)。本轮无需顺手修项——唯一的实质增量(注释修正)已由作者侧完成且核实正确。 |
(ai: vllm-qwen/qwen3.8-27b) review 第 2 轮完成(head aedb2bc):APPROVE,可合并。 增量 = master 合入 + 一个注释修正 commit debbae4(把「半开探测每 60s」纠正为「请求驱动、无背景定时器」——已核实为真:模块零定时器,探测逐请求咨询 relay.ts:810;纯注释零行为变更)。branch 对新 master 净 diff 仍严格 3 文件;新 head 本地定向 47/47 + typecheck 干净;全量 CI 写作时仍在排队(queued 14 / failed 0),绿灯以 CI 为准。另记录一个非阻塞观察:断路器表本身无老化,断路已开的被放弃 host 会经 /__bili/upstreams 的断路器条目继续显示——属 #3148 机制自身既有行为,是否加时效待 owner 判断,未在本 PR 处置。完整结论见 PR 评论。 |
📦 Merged Build PublishedThis merge to master was published as npm install -g billion-context@0.0.1025-master.ce2ae72 # this exact build
npm install -g billion-context@master # always the newest merged build
|
…ction (#5701) #5574 made alert rows age out after 5 min, but getUpstreamHealthReport merged every OPEN circuit unconditionally — a host that tripped its circuit and then stopped being hit rendered in the DSH banner forever (rows synthesized from openedAt/lastFailure.at), because circuit entries are only removed by a success or MAX_CIRCUITS eviction. Filter the projection only, mirroring pruneStaleAlerts: an open circuit whose last activity (lastFailure.at ?? openedAt) predates the same ALERT_STALE_MS bound drops out of /__bili/upstreams. The state machine keeps the entry — traffic resuming into it still fails fast / probes, and a fresh failure re-surfaces the row on the very next read. Both directions pinned in tests/upstream-alerts.test.ts.
Closes #5572
问题
DSH 桌面端 plugin-native 模式下,「上游不可用」横幅(shell.overlay 红条)在一次性 ECONNREFUSED 后会永久卡住,只能重启宿主才消失。典型触发:provider 是宿主内 loopback shim(
listen(0)随机端口),插件重载后旧 listener 关闭、新 listener 换到新的临时端口;已有会话里一次迟到的请求打到旧的死端口 → 一条connect-refused,之后流量走新端口完全正常,但横幅一直挂着。证据:count:1, circuit:null, firstSeen==lastSeen。根因
src/upstream-alerts.ts的告警表只有一条移除路径:对同一 host 成功 fetch(或进程重启)。死端口永远不会再成功 → 成功清除路径对它永不触发 → 该行永久驻留。断路器要 3 次连续失败才打开,count:1根本没碰到它。这张表的语义是「当前时刻的活跃快照」(#1682),却没有任何时效上界。修法
加一个时效上界
ALERT_STALE_MS = 5min:最后一次失败早于now − 5min的行,在两个只读投影里惰性剪掉——/__bili/upstreams(喂 DSH overlay 的健康报告)与/__bili/overview的 alerts(喂 web 横幅)。行是否存活只看lastSeen:仍被请求且仍在失败的 host 每次失败都刷新 lastSeen,所以真宕机期间一直可见;断路器半开探测是请求驱动的(每次发请求前查一次,无后台定时器),因此一个不再被任何会话引用的 host(模型切走 / provider 被移除)既不会有新失败也不会有探测,lastSeen 冻结——这正是 TTL 要清理的僵尸,也是它唯一的出口。不加任何配置面;上界统一(不专门针对 loopback),因为「按新鲜度判活」与 IP 段无关——现场已见真实用户配置的本地 OpenAI 兼容 provider(llama.cpp,会话切走模型后)同样永久卡死,印证了这不是 loopback-shim 专属。行为变更(旧 → 新)
范围外(待 owner 拍板,详见 issue 评论)
issue 提的另外两个方向本 PR 不做:② 会话级 loopback upstreamOrigin 失效/刷新(属 session-binding 领域,§7.4 人工门禁);③ 每行 dismiss/ack +
dsh.upstreamBanner配置开关(新增配置面,owner-gated)。注:web 横幅其实已有 per-instance dismiss,DSH overlay 没有——是否给 DSH overlay 也加 dismiss 属产品决策。验证
本地定向全绿:
tests/upstream-alerts.test.ts(新增 3 个回归:一次失败过期 / 仍在失败不老化 / 健康报告投影同样丢弃)、tests/issue5146-upstreams-endpoint.test.ts、tests/issue3148-upstream-circuit.test.ts、tests/issue4254-upstream-failure-egress.test.ts、tests/skipped-proxy-hint.test.ts;npm run typecheck干净。全量回归交给 CI。