Repository navigation
fix: fail fast on dead upstreams — round-level circuit breaker + acp_status visibility (#3148) - #3154
Conversation
…status visibility (#3148) A mid-session upstream outage was amplified per round: every host-driven retry burned the full in-request ladder (~4.5s) plus a full cold resend of the context, with no round-level memory that the endpoint was already proven dead. The compress path had a 3-strike breaker; the upstream path had none. - upstream-alerts.ts: per-host circuit — 3 consecutive FINAL pre-response transport failures (connect-refused/connect-timeout/dns/proxy-reset/upstream-reset; timeouts+aborts+5xx deliberately excluded) open it; any resolved response closes it; half-open admits exactly one probe after a 60s cooldown, a failed probe re-arms. - relay.ts forward(): while open, requests fail fast BEFORE any fetch with the last failure kind/detail/time and the next-probe ETA in the error (surfaces as the existing 502 detail — no status-code change); the fast-fail round still arms emergency shrink and drops attached retrievals like a network failure. - acp_status: new sections rendered only when non-trivial — UPSTREAM CIRCUIT OPEN, EMERGENCY SHRINK ARMED (marker now recorded at arm time and cleared by the next real usage report), OUTPUT BUDGET CLAMP (last clamp event + sticky #2490 planning floor). Display-only; no wire decision changes. No new config/env/CLI surface; breaker constants are internal (knob-ization owner-gated). Regression tests: tests/issue3148-upstream-circuit.test.ts (8 cases incl. the reporter's exact 65536->1360 clamp math).
📦 Built Package ArtifactBranch: Option A — Install this exact build (recommended)npm install -g billion-context@0.0.3603-pr.3154.10b0376This exact version is immutable and reproducible. The rolling Option B — Download artifact
tar xzf billion-context-pr3154.tgz
npm install -g packageInstall instructions are refreshed on each push. |
…th an uncounted kind (#3154 review)
(ai: vllm-qwen/qwen3.8-27b) 独立审核进行中,先报两个已当场修掉并推入 PR 分支的发现( 1. CI 红:unused-exports 门禁失败。 根因:本 PR 新增的 2. 熔断器自身的状态机缺陷(代码审出,非既有行为回归)。 半开探针以「不计入」种类的失败结束时(超时 / 客户端中止 / unknown——即不在 ALERT_KINDS 里的), 本地定向验证:unused-exports 门禁绿、 |
(ai: vllm-qwen/qwen3.8-27b) 独立审核完成(未照抄 #3148 侧结论,diff 逐文件重读、行为逐项清点)。审核中发现两处问题已当场修掉并推入分支( 1. 是什么问题:上游端点在会话中途不可用时,bili 没有轮级记忆——宿主每重试一轮,bili 都完整烧一遍请求内重试阶梯(~4.5s)再全量冷发一次上下文到已证明死掉的端点(#3148 现场:20 连击 ≈4.8MiB);同时熔断外的三种失败状态(紧急收缩 armed、输出钳制、上游失败)只存在于 bili.log,宿主/模型看不到「为什么这轮失败」。 2. 什么原因导致的:compress 路径早有 3 连败熔断,上游路径从未有过对等的轮级状态(每请求无状态是既有设计);可见性状态写在 warn 日志行里、没有落到 session.stats,acp_status 自然读不到。 3. 影响范围与严重程度:所有「上游中途失联 + 宿主会重试」的会话(dsh 模式最典型)。修前属体验/带宽放大问题而非正确性缺陷:不丢数据,但每轮多付 ~4.5s 延迟 + 完整 payload 重发,且用户无从判断端点是否真的死了。可见性缺口为诊断便利级别。 4. 怎么修复的:机制层两件事——① 按 host 的上游熔断:连续 3 次最终 pre-response 传输失败(口径与 #1682 ALERT_KINDS 完全一致,含 tls;不含超时/客户端中止/5xx)打开,打开期间 fetch 之前 fail-fast,错误走既有 502 detail 通道(状态码不变),带最后一次失败的 kind/detail/时间 + 下次探针时间;60s 冷却后半开单飞探针,任何已解析响应(含 5xx)立即关闭。② 纯展示层可见性:acp_status 三个条件段(UPSTREAM CIRCUIT OPEN / EMERGENCY SHRINK ARMED / OUTPUT BUDGET CLAMP),数据来自 session.stats 新增可选字段(armFailureShrink 写入 shrink marker、下一次真实 usage report 清除;clamp 经可选 hook 在既有日志行之后记录)——不触碰任何 wire 决策。 5. 是否有回归问题:审了 diff 触碰路径的全部既有行为,结论均为保持——正常失败路径错误串字节不变(failureDetail 同一 formatUpstreamError 结果);成功路径不变(+1 幂等关闭调用);armFailureShrink 抬升逻辑与 markDirty 不变(+2 附加字段);settleUsageReport 既有写入不变(+2 附加删除);clamp 数学不变(测试钉住 reporter 精确数字 65536→1360,hook 前后一致,健康场景零调用);acp_status 无状态时完全静默(clean fixture 钉住);持久化字段全部可选附加(旧文件不受影响);5xx passthrough、nudge 策略、#2490 sticky floor 语义、重试阶梯常量均未动。高危模式逐项过:无吞错(fail-fast 仍 throw 仍 502,detail 更明确)、无阈值/时序漂移、输出格式变化仅限披露的新增展示行与新失败路径的前缀 6. 是否破坏缓存:不涉及。前缀缓存/ledger/重放游标零改动;fail-fast 时的 dropRetrievals 与既有网络失败路径同一语义(附件全文没到过模型就丢弃并留痕)。 7. 是否改了设计方向:没有转向。是对既有告警/熔断家族(#1682 告警表 + compress-loop 熔断先例)的同构扩展;刻意不建配置面(阈值 3 / 冷却 60s 为内部常量,镜像 compress-loop breaker 同样无配置项的先例),PR 里已注明如需可配置化建议挂 network.* 下由 owner 拍板——同意这个处理。小瑕疵一处不拦:PR 正文列举计入种类时漏写了 tls(代码实际与 ALERT_KINDS 完全一致,含 tls,描述少列了一个词)。 8. 是否引入了新的配置:无。无新 env/config/默认值。 判定:可合并。 head |
📦 Merged Build PublishedThis merge to master was published as npm install -g billion-context@0.0.433-master.bf92293 # this exact build
npm install -g billion-context@master # always the newest merged build
|
…ssed by #3101's re-pin) #3154 (merged 2026-10-15 09:35) pinned the #3148 clamp event at capped=1360. #3101 (merged 2026-10-15 20:30) introduced the owner-signed #3071 floor floor_out=min(4096, window-input) and re-pinned the pre-existing #2122/#2558 pins, but its re-pin commit predates #3154, so the newer #3148 pin stayed stale and every CI test shard touching this file has been red since. Both asserts (event.capped and rebuilt.max_tokens) come from the same floored value in clampOutgoingOutput; headroom 51228 >= 4096 so the floored value is 4096. The tooth still bites: the scenario still clamps 65536 far below the requested budget.
Closes #3148
(ai: vllm-qwen/qwen3.8-27b)
修什么
#3148 证实:上游在会话中途不可用时,bili 没有「轮级记忆」——dsh 每重试一轮,bili 都完整烧一遍请求内重试阶梯(~4.5s)再全量冷发一次上下文;20 连击 ≈4.8MiB 全部发往已证明死掉的端点。compress 路径早有 3 连败熔断,上游路径没有;紧急收缩、输出钳制、最后一次上游失败三种状态也只存在于 bili.log,acp_status 看不到。
两处改动(机制层,无新配置项):
1. 上游熔断(src/upstream-alerts.ts + src/server/relay.ts)
2. 失败状态可见性(src/acp-status.ts + session.ts + cache-ledger.ts + budget.ts + prepare-{openai,responses,google}.ts)
UPSTREAM CIRCUIT OPEN(host、连续失败数、起始时间、最后失败详情、下次探针时间);EMERGENCY SHRINK ARMED(估计值 + arm 时间;marker 在 arm 时写入、下一次真实 usage report 清除);OUTPUT BUDGET CLAMP(最近一次钳制事件 raw→capped/window/input + sticky planning floor dsh压缩与bili压缩冲突 #2490)。行为变更(旧 → 新)
upstream request failed: …upstream circuit open: <host>: N consecutive pre-response failures … next probe <ISO> — stop resending full context开头不变:请求内重试阶梯(network.replayRetryMax/BaseMs)、5xx passthrough、所有状态码、#2490 sticky floor 语义、nudge 策略。
config surface
未新增。熔断阈值(3)与冷却(60s)为内部常量,镜像 compress-loop breaker 先例(同样无配置项)。如需可配置化,建议 owner 后续拍板形态(如挂 network.* 下),本 PR 不擅自建形状。
测试
相关但不在本 PR 范围(见 issue 回复)