Skip to content

[PR] Make upstream-failure egress diagnosable (#4254): informative error, 503+Retry-After on circuit-open - #4257

Closed
ranxianglei wants to merge 4 commits into
masterfrom
2026-10-22_4254-upstream-failure-egress
Closed

ranxianglei wants to merge 4 commits into
masterfrom
2026-10-22_4254-upstream-failure-egress

Conversation

@ranxianglei

@ranxianglei ranxianglei commented Oct 22, 2026 •

Copy link
Copy Markdown
Owner

Addresses ranxianglei/billion-context#4254 — defects 1, 3, 4, 5, 6, plus the bili-side read surface for defect 2 (GET /__bili/upstreams). Defect 2's DSH client-side banner is cross-boundary (DSH slot rendering) and tracked in-thread.

What was wrong

The client renders the error string of the proxy-failure response, but bili answered every failure with the constant "acp-proxy failure" while the real cause already lived in detail, the structured upstreamFailure{kind,code,hint}, and the alerts[] table. On a dead / circuit-open upstream the DSH client saw only those six words.

Changes

  1. Informative error (defect 1). New shared builder buildProxyFailureResponse() used by BOTH egress sites (startServer dispatch catch + the inline engine, which previously omitted the structured field entirely). When a failure carries a structured slice, error reads "acp-proxy failure: <kind> [<code>] [host[:port]]" (+ a (circuit open, next probe +Ns) suffix on fail-fast). The legacy prefix is preserved verbatim and the shape stays a string (not the OpenAI {error:{message}} object form, which would break hosts that render error.text).
  2. Circuit-open fail-fast → 503 + Retry-After (defect 6). A round-level breaker fail-fast now answers 503 with Retry-After set to the next-probe interval, so compliant hosts back off instead of resending full context. Precedent: [Evidence] pi 车道切供应商(同 modelId)后 fold 不随会话迁移:裸上下文 2.23MB(≥298K est tokens)被 preflight 10 秒拒 3 次;账面 ≤157K 与裸水位背离 10.5h #2847 moved doomed overflow 502→413 for the same reason. Initial transport failures stay 502; overflow stays 413.
  3. Log amplification (defect 3). While a circuit stays OPEN, the per-request [warn] is throttled to one line/window/host and the dispatch-catch [error]+stack is suppressed for expected fail-fast (non-enumerable marker). State flips (OPEN/CLOSED/probe) still log at their own sites.
  4. Masking consistency (defect 4). The free-text circuit reason now masks its host through the same rule as the embedded last-failure detail (non-public → <private-host>), ending the raw-next-to-masked contradiction. The actionable endpoint rides the structured upstreamFailure.host field (loopback-exempt). Free-text masking is otherwise unchanged — 探索 daemon 卡死时 acp 会话的恢复路径 #255 stays green.
  5. Word-boundary truncation (defect 5). Circuit last-failure detail truncates at the last whitespace before the cap instead of mid-token (hint=TCP re…).
  6. Upstream-health read surface (defect 2, bili half). New loopback-gated GET /__bili/upstreams projects the per-host alert table + open circuits into a stable contract {upstreams:[{host,port,kind,code,hint,count,firstSeen,lastSeen,circuit:{state,nextProbeInMs}}]} for DSH's frame-level banner. It adds an optional errno code to UpstreamAlert/UpstreamCircuitFailure so the cause survives to the GUI. Additive and inert until a client polls it; the open circuit is the authoritative "currently down" signal (alerts go stale once fail-fast stops minting them). Pairs with the in-conversation informative error from defect 1: the frame-level banner says which upstream is down, the per-request error says why yours failed this time.

Behavior changes (old → new)

Area Old New Why
error field constant "acp-proxy failure" "acp-proxy failure: <kind> [<code>] [host[:port]]" (+ circuit suffix); prefix kept, still a string clients read error
circuit-open status 502 503 + Retry-After (initial failures still 502, overflow still 413) backoff semantics (#2847)
OPEN-state logs [warn]+[error] per request ≤1 [warn]/window/host; per-request [error] suppressed stop amplification
free-text circuit host raw (e.g. 127.0.0.1:7864) masked <private-host> (consistent with embedded detail); actionable host in structured upstreamFailure.host one message, one masking regime (#255 intact)
circuit detail truncation hard cut at 240 chars back off to last whitespace no mid-word cut
upstream health read surface (none) GET /__bili/upstreams (loopback-gated): per-host merge of alerts + open circuits stable contract for the DSH frame-level banner (defect 2)

The operator-facing /__bili/acp panel is unaffected — it reads the raw state.host directly.

Config surface

None — no new env var / config key / CLI flag. The new GET /__bili/upstreams is a loopback-gated operator diagnostic endpoint (behind the same gate as /__bili/overview), not user-facing configuration.

Tests

New regression suite tests/issue4254-upstream-failure-egress.test.ts pins every facet (informative error + prefix, 503+Retry-After vs the 502-vs-413 boundaries, loopback-exempt display, word-boundary truncation, Retry-After ETA, fail-fast log throttle, marker non-enumerability). New tests/issue4254-upstreams-endpoint.test.ts pins the projection (port parsing incl. null-port hosts, circuit-authoritative down signal + nextProbe ETA, multi-host merge, recovery drops the host). Two existing pins updated where their asserted value IS the intentional change above (issue3776 exact-match error; issue3148 raw private host in the reason). Verified on the rebased head: typecheck clean, targeted suites green.

@ranxianglei ranxianglei changed the title Make upstream-failure egress diagnosable (#4254): informative error, 503+Retry-After on circuit-open [PR] Make upstream-failure egress diagnosable (#4254): informative error, 503+Retry-After on circuit-open Oct 22, 2026
@github-actions

github-actions Bot commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-22_4254-upstream-failure-egress (834de47)

Option A — Install this exact build (recommended)

npm install -g billion-context@0.0.5235-pr.4257.fbb0142

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr4257.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重: 不重复。平台与 GitHub 两侧检索(按失败签名 "acp-proxy failure" / upstream-failure egress / circuit-open 503)均无其他在途条目;相关历史项 #3148(熔断机制本身)、#3776(TLS 诊断)、#2847(413 先例)均已合并且本 PR 只在其之上改 egress 渲染,无并列修复。

标签: bug + dsh + scope/wiring + p2(上游故障时客户端只见六个字、OPEN 态日志放大、自由文本主机名裸奔——故障场景下的明显问题,有面板/日志绕行)。

这是带 ework-agent-pr 标记的 [PR] 单,不重做。正在做 PR 审核:分支已拉取核对(单 commit 64a3a1b,基于最新 master)、CI 静态门禁全绿、测试分片在跑;本地定向跑了 issue4254/issue3148/log-mask/main-path-transport-retry/e2e-relay-5xx-selfheal/issue3776 六个套件全绿。最终结论(结构化八问)稍后补上。

@ranxianglei ranxianglei added bug Something isn't working dsh p2 scope/wiring labels Oct 22, 2026
awork added 2 commits October 22, 2026 13:17
The client renders the `error` string, but bili answered every proxy failure
with the constant "acp-proxy failure" while the real cause already sat in
detail / upstreamFailure / alerts. Changes:

- buildProxyFailureResponse() shared by both egress sites (startServer dispatch
  catch + inline engine): `error` is now informative ("acp-proxy failure: <kind>
  [<code>] [host[:port]]", legacy prefix kept, stays a string); a round-level
  circuit-breaker fail-fast answers 503 + Retry-After (next-probe interval)
  instead of a generic 502. Initial transport failures stay 502; overflow 413.
- structured upstreamFailure gains host/circuitOpen/retryAfterSeconds (additive,
  byte-identical when absent); the inline lane now emits it too (was omitted).
- circuit-open fail-fast logs throttled to one [warn]/window/host; the per-request
  [error]+stack is suppressed via an expected-fail-fast marker (stops the storm).
- free-text circuit reason masks the host through the same rule as the embedded
  last-failure detail (non-public -> <private-host>), ending the raw-next-to-masked
  contradiction; the actionable endpoint rides structured upstreamFailure.host
  (loopback-exempt). Free-text masking otherwise unchanged (#255 intact).
- circuit last-failure detail truncates at a whitespace boundary, not mid-token.

Addresses #4254 (defects 1,3,4,5,6; defect 2 pending
a design decision tracked in-thread). Regression: tests/issue4254-*.test.ts.
Loopback-gated GET /__bili/upstreams projects the per-host alert table plus open circuits into a stable {upstreams:[{host,port,kind,code,hint,count,firstSeen,lastSeen,circuit:{state,nextProbeInMs}}]} contract for DSH's frame-level banner. Additive inert read surface: nothing changes until a client polls it. Adds optional errno code to UpstreamAlert/UpstreamCircuitFailure so the cause survives to the GUI.
awork added 2 commits October 23, 2026 17:44
…4254 review nit)

The suite's marker test asserted JSON.stringify(err) === '{}', but a symbol
key never reaches JSON.stringify whether enumerable or not, so the assertion
pinned nothing about the implementation choice. Pin the property descriptors
instead: every own property (incl. the marker) must stay non-enumerable.
…ort gate (#4254)

UpstreamHealthEntry/UpstreamHealthReport are the wire-shape interfaces for
the new loopback-gated GET /__bili/upstreams report. Like LastInstallRecord
(#3754) they are referenced only at their definition site — admin.ts and the
endpoint tests consume them structurally through getUpstreamHealthReport() —
so keep the exports named where the contract is defined.
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Review progress update — the branch moved while I was mid-review (rebase onto current master + new commit 20df7ea87 exposing GET /__bili/upstreams, the defect-② read surface). I reviewed the full current head 834de476e (all three commits) and pushed two small review fixes to the branch:

  1. fb34126c2 (test) — my earlier marker test pin was toothless: it asserted JSON.stringify(err) === "{}", but symbol keys never reach JSON.stringify whether enumerable or not, so it pinned nothing. Now pins by property descriptor: every own property of the marked error (incl. the marker) must be non-enumerable.
  2. 834de476e (gate) — the new endpoint's report interfaces (UpstreamHealthEntry/UpstreamHealthReport) tripped the unused-exports gate on 20df7ea87. Added a reasoned WAIVERS entry following the exact LastInstallRecord ([no-dispatch] Pinned installs get surprise reinstalls: advisory force-install ignores autoUpdate:false, and config-file changes only take effect on restart #3754) precedent: wire-shape interfaces referenced only at their definition site, consumed structurally via getUpstreamHealthReport(), kept named where the contract is defined.

Local verification on the final head: typecheck clean, unused-exports gate clean, 66 targeted tests green (issue4254 both suites, issue3148, log-mask, main-path-transport-retry, e2e-relay-5xx-selfheal, issue3776). Full-suite verdict = remote CI, currently running on 834de476e (static gates in flight, 0 failures so far). Full structured verdict follows as soon as CI lands.

yanhuojiyusheng pushed a commit to yanhuojiyusheng/billion-context that referenced this pull request Oct 24, 2026
Defect 2 render half of ranxianglei#4254, pairing with the GET /__bili/upstreams read surface shipped in ranxianglei#4257. bili's DSH client registers a frame-level shell.overlay slot that polls /__bili/upstreams every 10s and renders one row per unreachable upstream (host:port, kind, errno code, consecutive-failure tally, and a retry ETA while the circuit is open); it returns null while every upstream is reachable, so the banner auto-dismisses when circuits close + alerts clear.

- Display identity is the forwarded host:port (bili is a transparent pipe); provider/model reverse lookup stays host-side (DSH reads its own baseURL).

- Defensive parse of the report never throws: a broken health report cannot break the DSH client.

- Purely additive UI, no request-path changes; deferred injection is fail-open on hosts whose build lacks the shell.overlay slot.

- i18n zh/en in sync; deterministic render test (populated / healthy / multi-host).

(cherry picked from commit dce1538)
@ranxianglei

Copy link
Copy Markdown
Owner Author

(ai: zhipuai-lb/glm-5.3)

已并入合并列车 #4551(master 合并,merge commit 保留原作者署名)。本 PR 关单收尾;回滚点为 #4551 单提交 revert。

Louie0806 pushed a commit to Louie0806/billion-context that referenced this pull request Oct 31, 2026
…s its data (ranxianglei#5146)

The DSH frame-level upstream banner (shipped via 6874e39 from ranxianglei#4254) polls
/__bili/upstreams every 10s, but its server half never landed: PR ranxianglei#4257 closed
unmerged and the re-land (050241c) kept only the egress fixes. Every poll
therefore 404'd AND was CORS-blocked in the DSH renderer (admin JSON routes
send no Access-Control-Allow-Origin), so the banner never showed and each
session logged one console error per 10s.

- Add loopback-gated GET /__bili/upstreams to handleAdminRoute projecting the
  alert table + open circuits into the ranxianglei#4254-settled contract
  {upstreams:[{host,port,kind,code,hint,count,firstSeen,lastSeen,circuit}]};
  one row per host, the OPEN circuit outranks alerts (fail-fast stops minting
  alerts once open, so it is the authoritative down signal), alerts cover the
  pre-circuit window and carry the hint.
- Echo the requesting Origin as ACAO on this route only — same gate-first
  pattern as the /__bili/ shell and /__bili/plugin/status (ranxianglei#3862); only
  origins the trusted-origin gate admitted reach dispatch, so the echo cannot
  widen the admin surface. No Origin (curl/probes) → no ACAO, unchanged.
- Capture the errno at failure time into UpstreamAlert/UpstreamCircuitFailure
  (optional code field) so the report shows WHY beyond the classified kind —
  the client banner renders it when present.
- Tests: projection semantics (alert-only window / circuit precedence /
  recovery drops the host / portless host) plus the HTTP surface (ACAO echo,
  healthy empty list without ACAO, non-loopback 403, tunnel 403).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working dsh p2 scope/wiring

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant