Repository navigation
[PR] Make upstream-failure egress diagnosable (#4254): informative error, 503+Retry-After on circuit-open - #4257
ranxianglei wants to merge 4 commits into
Conversation
📦 Built Package ArtifactBranch: Option A — Install this exact build (recommended)npm install -g billion-context@0.0.5235-pr.4257.fbb0142This exact version is immutable and reproducible. The rolling Option B — Download artifact
tar xzf billion-context-pr4257.tgz
npm install -g packageInstall instructions are refreshed on each push. |
(ai: vllm-qwen/qwen3.8-27b) 查重: 不重复。平台与 GitHub 两侧检索(按失败签名 "acp-proxy failure" / upstream-failure egress / circuit-open 503)均无其他在途条目;相关历史项 #3148(熔断机制本身)、#3776(TLS 诊断)、#2847(413 先例)均已合并且本 PR 只在其之上改 egress 渲染,无并列修复。 标签: bug + dsh + scope/wiring + p2(上游故障时客户端只见六个字、OPEN 态日志放大、自由文本主机名裸奔——故障场景下的明显问题,有面板/日志绕行)。 这是带 |
The client renders the `error` string, but bili answered every proxy failure
with the constant "acp-proxy failure" while the real cause already sat in
detail / upstreamFailure / alerts. Changes:
- buildProxyFailureResponse() shared by both egress sites (startServer dispatch
catch + inline engine): `error` is now informative ("acp-proxy failure: <kind>
[<code>] [host[:port]]", legacy prefix kept, stays a string); a round-level
circuit-breaker fail-fast answers 503 + Retry-After (next-probe interval)
instead of a generic 502. Initial transport failures stay 502; overflow 413.
- structured upstreamFailure gains host/circuitOpen/retryAfterSeconds (additive,
byte-identical when absent); the inline lane now emits it too (was omitted).
- circuit-open fail-fast logs throttled to one [warn]/window/host; the per-request
[error]+stack is suppressed via an expected-fail-fast marker (stops the storm).
- free-text circuit reason masks the host through the same rule as the embedded
last-failure detail (non-public -> <private-host>), ending the raw-next-to-masked
contradiction; the actionable endpoint rides structured upstreamFailure.host
(loopback-exempt). Free-text masking otherwise unchanged (#255 intact).
- circuit last-failure detail truncates at a whitespace boundary, not mid-token.
Addresses #4254 (defects 1,3,4,5,6; defect 2 pending
a design decision tracked in-thread). Regression: tests/issue4254-*.test.ts.
Loopback-gated GET /__bili/upstreams projects the per-host alert table plus open circuits into a stable {upstreams:[{host,port,kind,code,hint,count,firstSeen,lastSeen,circuit:{state,nextProbeInMs}}]} contract for DSH's frame-level banner. Additive inert read surface: nothing changes until a client polls it. Adds optional errno code to UpstreamAlert/UpstreamCircuitFailure so the cause survives to the GUI.
0ea3e74 to
20df7ea
Compare
…4254 review nit) The suite's marker test asserted JSON.stringify(err) === '{}', but a symbol key never reaches JSON.stringify whether enumerable or not, so the assertion pinned nothing about the implementation choice. Pin the property descriptors instead: every own property (incl. the marker) must stay non-enumerable.
…ort gate (#4254) UpstreamHealthEntry/UpstreamHealthReport are the wire-shape interfaces for the new loopback-gated GET /__bili/upstreams report. Like LastInstallRecord (#3754) they are referenced only at their definition site — admin.ts and the endpoint tests consume them structurally through getUpstreamHealthReport() — so keep the exports named where the contract is defined.
(ai: vllm-qwen/qwen3.8-27b) Review progress update — the branch moved while I was mid-review (rebase onto current master + new commit
Local verification on the final head: typecheck clean, unused-exports gate clean, 66 targeted tests green (issue4254 both suites, issue3148, log-mask, main-path-transport-retry, e2e-relay-5xx-selfheal, issue3776). Full-suite verdict = remote CI, currently running on |
Defect 2 render half of ranxianglei#4254, pairing with the GET /__bili/upstreams read surface shipped in ranxianglei#4257. bili's DSH client registers a frame-level shell.overlay slot that polls /__bili/upstreams every 10s and renders one row per unreachable upstream (host:port, kind, errno code, consecutive-failure tally, and a retry ETA while the circuit is open); it returns null while every upstream is reachable, so the banner auto-dismisses when circuits close + alerts clear. - Display identity is the forwarded host:port (bili is a transparent pipe); provider/model reverse lookup stays host-side (DSH reads its own baseURL). - Defensive parse of the report never throws: a broken health report cannot break the DSH client. - Purely additive UI, no request-path changes; deferred injection is fail-open on hosts whose build lacks the shell.overlay slot. - i18n zh/en in sync; deterministic render test (populated / healthy / multi-host). (cherry picked from commit dce1538)
…s its data (ranxianglei#5146) The DSH frame-level upstream banner (shipped via 6874e39 from ranxianglei#4254) polls /__bili/upstreams every 10s, but its server half never landed: PR ranxianglei#4257 closed unmerged and the re-land (050241c) kept only the egress fixes. Every poll therefore 404'd AND was CORS-blocked in the DSH renderer (admin JSON routes send no Access-Control-Allow-Origin), so the banner never showed and each session logged one console error per 10s. - Add loopback-gated GET /__bili/upstreams to handleAdminRoute projecting the alert table + open circuits into the ranxianglei#4254-settled contract {upstreams:[{host,port,kind,code,hint,count,firstSeen,lastSeen,circuit}]}; one row per host, the OPEN circuit outranks alerts (fail-fast stops minting alerts once open, so it is the authoritative down signal), alerts cover the pre-circuit window and carry the hint. - Echo the requesting Origin as ACAO on this route only — same gate-first pattern as the /__bili/ shell and /__bili/plugin/status (ranxianglei#3862); only origins the trusted-origin gate admitted reach dispatch, so the echo cannot widen the admin surface. No Origin (curl/probes) → no ACAO, unchanged. - Capture the errno at failure time into UpstreamAlert/UpstreamCircuitFailure (optional code field) so the report shows WHY beyond the classified kind — the client banner renders it when present. - Tests: projection semantics (alert-only window / circuit precedence / recovery drops the host / portless host) plus the HTTP surface (ACAO echo, healthy empty list without ACAO, non-loopback 403, tunnel 403).
Addresses ranxianglei/billion-context#4254 — defects 1, 3, 4, 5, 6, plus the bili-side read surface for defect 2 (
GET /__bili/upstreams). Defect 2's DSH client-side banner is cross-boundary (DSH slot rendering) and tracked in-thread.What was wrong
The client renders the
errorstring of the proxy-failure response, but bili answered every failure with the constant"acp-proxy failure"while the real cause already lived indetail, the structuredupstreamFailure{kind,code,hint}, and thealerts[]table. On a dead / circuit-open upstream the DSH client saw only those six words.Changes
error(defect 1). New shared builderbuildProxyFailureResponse()used by BOTH egress sites (startServerdispatch catch + the inline engine, which previously omitted the structured field entirely). When a failure carries a structured slice,errorreads"acp-proxy failure: <kind> [<code>] [host[:port]]"(+ a(circuit open, next probe +Ns)suffix on fail-fast). The legacy prefix is preserved verbatim and the shape stays a string (not the OpenAI{error:{message}}object form, which would break hosts that rendererror.text).Retry-After(defect 6). A round-level breaker fail-fast now answers 503 withRetry-Afterset to the next-probe interval, so compliant hosts back off instead of resending full context. Precedent: [Evidence] pi 车道切供应商(同 modelId)后 fold 不随会话迁移:裸上下文 2.23MB(≥298K est tokens)被 preflight 10 秒拒 3 次;账面 ≤157K 与裸水位背离 10.5h #2847 moved doomed overflow 502→413 for the same reason. Initial transport failures stay 502; overflow stays 413.[warn]is throttled to one line/window/host and the dispatch-catch[error]+stack is suppressed for expected fail-fast (non-enumerable marker). State flips (OPEN/CLOSED/probe) still log at their own sites.non-public → <private-host>), ending the raw-next-to-masked contradiction. The actionable endpoint rides the structuredupstreamFailure.hostfield (loopback-exempt). Free-text masking is otherwise unchanged — 探索 daemon 卡死时 acp 会话的恢复路径 #255 stays green.hint=TCP re…).GET /__bili/upstreamsprojects the per-host alert table + open circuits into a stable contract{upstreams:[{host,port,kind,code,hint,count,firstSeen,lastSeen,circuit:{state,nextProbeInMs}}]}for DSH's frame-level banner. It adds an optional errnocodetoUpstreamAlert/UpstreamCircuitFailureso the cause survives to the GUI. Additive and inert until a client polls it; the open circuit is the authoritative "currently down" signal (alerts go stale once fail-fast stops minting them). Pairs with the in-conversation informativeerrorfrom defect 1: the frame-level banner says which upstream is down, the per-request error says why yours failed this time.Behavior changes (old → new)
errorfield"acp-proxy failure""acp-proxy failure: <kind> [<code>] [host[:port]]"(+ circuit suffix); prefix kept, still a stringerrorRetry-After(initial failures still 502, overflow still 413)[warn]+[error]per request[warn]/window/host; per-request[error]suppressed127.0.0.1:7864)<private-host>(consistent with embedded detail); actionable host in structuredupstreamFailure.hostGET /__bili/upstreams(loopback-gated): per-host merge of alerts + open circuitsThe operator-facing
/__bili/acppanel is unaffected — it reads the rawstate.hostdirectly.Config surface
None — no new env var / config key / CLI flag. The new
GET /__bili/upstreamsis a loopback-gated operator diagnostic endpoint (behind the same gate as/__bili/overview), not user-facing configuration.Tests
New regression suite
tests/issue4254-upstream-failure-egress.test.tspins every facet (informative error + prefix, 503+Retry-After vs the 502-vs-413 boundaries, loopback-exempt display, word-boundary truncation, Retry-After ETA, fail-fast log throttle, marker non-enumerability). Newtests/issue4254-upstreams-endpoint.test.tspins the projection (port parsing incl. null-port hosts, circuit-authoritative down signal + nextProbe ETA, multi-host merge, recovery drops the host). Two existing pins updated where their asserted value IS the intentional change above (issue3776exact-matcherror;issue3148raw private host in the reason). Verified on the rebased head: typecheck clean, targeted suites green.