Skip to content

fix: external summary credentials — readable failures, bounded retry, tri-state status (#3488) - #3541

Merged
ranxianglei merged 5 commits into
masterfrom
2026-10-17_external-summary-credentials-retry
Oct 18, 2026
Merged

ranxianglei merged 5 commits into
masterfrom
2026-10-17_external-summary-credentials-retry

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

(ai: vllm-qwen/qwen3.8-27b)

Closes #3488

Sub-item 1 of #3488 (PUT /__bili/external-summary/credential never registered → 404) is already fixed by open PR #3445 (closes #3443) — this PR does not touch that registration and stands independently for sub-items 2–4.

What was broken

  • Every failed external-summary attempt collapsed to a bare error outcome: no HTTP status, no response body anywhere. A single 429/5xx killed the whole batch (reason=exhausted, 0 folds) with nothing actionable in the log.
  • The credential store tri-state (not set / set-but-unusable / store unreadable) was flattened to one boolean, so the Web UI showed 未配置 for all three and a whitespace-polluted env: value could only be discovered at first dial.

Changes

Per-attempt logging + bounded retry (src/external-summary-http.ts)

  • Each failed dispatch logs locally: HTTP <status> from the masked host plus a whitespace-collapsed ≤1 KB response-body snippet (streaming-capped read — a hostile multi-MB error body is not buffered whole); pre-response network failures log the classified kind (connect-refused, upstream-timeout, …).
  • Bounded retry reuses the main chain's existing verdicts: isTransientUpstreamError (429 / ≥5xx / risk-control body markers) for HTTP responses and isFailFastUpstreamKind for network errors. Up to 3 dispatches per candidate, exponential backoff 1 s → 2 s (cap 8 s), Retry-After honored capped at 15 s. Non-transient statuses still fail fast on the first response. The whole loop stays inside the executor's existing per-attempt deadline (targetTimeoutMs), so the worst-case wall-clock bound is unchanged.
  • Internal constants only — no new config keys or env vars.

Tri-state credential status (src/external-summary-credentials.ts, src/web/api.ts, src/web/client.ts, src/web/i18n.ts)

  • New five-state model: configured | missing | invalid_value | store_unreadable | invalid_reference, with human-readable descriptions per state.
  • GET /__bili/config gains an ADDITIVE field externalSummaryCredentialStatus (reference → state). The legacy boolean externalSummaryCredentials is byte-for-byte unchanged.
  • Web UI badge now distinguishes 未配置 / 值不可用 / 存储不可读 / 引用无效 instead of everything rendering as unconfigured; Save-Key errors surface the server message instead of [object Object] (json() now extracts .message from object error bodies).

Boot-time audit (src/server.ts) — at startup, every effective enabled external-summary chain (global → route → model cascade, deduped) is checked; missing/unusable credential references produce one warn line each, e.g. [config] external summary credential: target nexus--free: environment variable NEXUS_API_KEY holds an unusable value (…). No new behavior knobs.

Behavior changes (old → new)

Area Old New
Candidate dispatch exactly 1 attempt per candidate up to 3 on transient errors (429/5xx/risk markers/fail-fast network); non-transient still fails fast immediately
Failure visibility attempts=[t0:error] only per-attempt local log lines with HTTP status + masked host + ≤1 KB body snippet (never leaked to results/wire)
Config GET externalSummaryCredentials: {ref: boolean} same booleans PLUS additive externalSummaryCredentialStatus: {ref: five-state}
UI badge configured / 未配置 5 states incl. unusable value, unreadable store, bad reference
Startup silent until first dial warn line per unresolved/unusable reference in enabled chains

No wire format change, no persistence change, no kernel change, no new dependencies.

Config surface

No new config fields, env vars, or CLI flags. One additive read-only field on an existing admin GET endpoint (legacy consumers unaffected). Nothing to migrate.

Tests

  • tests/external-summary-http.test.ts: split the old status loop into fail-fast (400/401/403 → primary+backup once, zero backoffs) vs transient (429/500/503 → 3 primary attempts with recorded 1000/2000 ms backoffs + backup, three logged rejection lines, result stays leak-free); new tests for Retry-After recovery, risk-control 403 marker retry, socket-death retry, exhausted-chain logging.
  • tests/issue3488-credential-status.test.ts (new): five-state matrix for env refs, malformed refs, secret lifecycle incl. corrupt-file and 0o644-permission paths, auditExternalSummaryCredentials against named recipes, boot-audit cascade over global/route/model, and handleConfigGet asserting the additive field alongside the untouched legacy booleans.
  • tests/external-summary-runtime.test.ts: two assertions updated to the new ≤3-attempts contract (commented), plus the retry-delay test seam.

Local on rebased head: typecheck clean; 109/109 across the four external-summary suites + 67/67 adjacent (web-routing, lane-credentials, summary badge/batch/autofold/executor-parallel). Full suite = CI.

Note: hot-file coordination with open #3460 (fold storm) — both touch src/server.ts; merged cleanly here.

@github-actions

github-actions Bot commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-17_external-summary-credentials-retry (b012950)

Option A — Install this exact build (recommended)

npm install -g billion-context@0.0.4213-pr.3541.8a57d94

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr3541.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

收到,开始审阅本 PR(不重做,核对代码 + CI + 定向测试)。

查重: 不重复。 检索 open+closed 的 issue/PR 后确认:在途 PR 中只有本 PR 触碰 external-summary 凭证链路的失败可见性/重试/三态状态。相关但不重复的:

当前状态:mergeable=true,但分支 behind master(需 rebase);CI 状态与代码审查进行中,结论稍后按八问体例回复。

awork added 5 commits October 17, 2026 19:37
…sk urls in logged snippets

Review fixes on #3541 (two small defects in the new retry loop):
- waitBeforeRetry pushed every scheduled delay into the module-level
  scheduledSummaryRetryDelaysMs array even in production (override null),
  so long-lived processes leaked entries on every transient retry. The
  recording now only happens when the test override is armed.
- the HTTP failure body snippet was logged raw while the pre-response
  path masks its error text; maskUrlsInText now applies to the log line
  only (the transient verdict below still needs the raw body markers).
…ad of mislabeling

The five-state badge expression left badgeState undefined when no detail
entry existed AND the legacy boolean was false — reachable for recipes
without credentialRef/apiKeyEnv (agent-registered #2336 providers) on NEW
servers, rendering 'Invalid reference format' where the pre-PR UI showed
'Not configured'. Fallback now maps absent detail to configured/missing
from the legacy record.
…inct from backoff

retry-after: '1' computed to 1000 ms — identical to the attempt-1 backoff
fallback — so the [1000] assertion could not tell a honored header from an
ignored one. Now uses '7' (asserts [7000]) and adds a hostile '999' case
pinning the 15 s cap.
…heir owning surfaces

The startup audit expanded chains against FILE recipes only, but request
time also sees agent-registered recipes (#2336, in-memory, registered
after boot). A chain targeting an agent-layer provider therefore emitted
a false 'unknown provider' warn at every start although it resolves fine
at request time. Unresolvable references are already surfaced by web save
(400) and the request-path one-shot warn, so the audit now reports only
what only it can know: credential states of expandable chains.
@ranxianglei
ranxianglei force-pushed the 2026-10-17_external-summary-credentials-retry branch from 23253a5 to b012950 Compare October 17, 2026 11:39
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

审核结论:通过,可合并(合并由人来)

分支已 rebase 到当前 master(v0.1.198,与 master 零文件重叠,rebase 干净),并补了 4 个 review 小修 commit(见下)。新 head b0129506d 上 CI 48/48 全绿(含此前 flake 的两个 Windows 22 shard——本次一次通过;上一个 head 23253a5 的 Windows 红是已知时序 flake #3530/#3478/#3554,master tip 自身在 Windows 22 也是红的,与本 PR 内容无关,未另立 issue)。本地定向验证:typecheck 干净 + 三个受影响套件 67/67 过。

结构化八问

  1. 是什么问题:外部摘要凭证三条失败路径的体验/可诊断性缺陷——凭证缺失或上游拒绝时用户只看到不透明报错;瞬态失败一次即判死、无重试;配置页只有「已配置/未配置」两态,分不清「值无效」和「没配」。
  2. 什么原因导致的:原实现每次尝试只发一次请求、失败不分类(HTTP 状态 vs 网络预响应错误 vs 非瞬态一概而论);凭证状态建模成单一布尔值,没有「值有效性/store 可读性/引用有效性」的状态空间;Web 层把 Error 对象直接拼进 DOM([object Object])。
  3. 影响范围与严重程度:仅影响启用 externalSummary 链且遇到凭证/上游失败的会话,默认用户不受影响。严重度 = 可诊断性缺陷 + 体验瑕疵(发生时摘要降级/失败且无从定位原因),不影响核心功能可用性。
  4. 怎么修复的:① 逐次尝试分类日志(HTTP 状态 + 脱敏 host + ≤1KB 折叠正文片段;预响应失败记分类 kind);② 有界重试复用主链既有裁决(isTransientUpstreamError / isFailFastUpstreamKind),每候选最多 3 次派发、退避 1s→2s 上限 8s、Retry-After 上限 15s、非瞬态立即失败——我核实了 executor 的 abort(targetTimeoutMs)包住整个重试循环,墙钟时间不会因重试变长;③ 五态凭证模型 configured|missing|invalid_value|store_unreadable|invalid_reference;GET /__bili/config 只加 externalSummaryCredentialStatus,旧字段字节级不变(我验证了 status() 与 resolve()/configured() 对三种 ref 形态的布尔等价);UI 五态徽标 + Save-Key 错误显示 .message。
  5. 是否有回归问题:逐项审过既有行为——legacy 布尔字段字节不变(等价性已验);旧报错字符串零消费者(grep 确认);重试不突破 targetTimeoutMs 边界;启动审计的级联展开与运行时 mergeCompress 整链替换语义一致(无假阳/假阴)。未发现回归。本轮另修 4 个小问题,各自独立 commit:
    • 057aab1e7 重试延迟记录缝只在测试模式生效(此前生产环境频繁瞬态重试会无限增长模块级数组)+ 日志片段做 URL 脱敏;
    • 1d30702b7 徽标兜底:detail 缺失且 legacy=false 时显示「未配置」而非误显「引用格式无效」;
    • 5f649d700 Retry-After 测试原来用 "1"(== 退避回退值 1000ms,断言无法区分是否真的读了头部,恒真风险),改 "7"→断言 [7000],并加 "999"→15s 封顶用例;
    • b0129506d 启动审计对启动时不可解析的引用不再报 warn(agent 层 recipe 在 boot 后注册,boot 时必然假阳)——该类引用问题仍由 web save 400 和请求路径一次性 warn 覆盖,测试断言同步翻转并注明理由。
  6. 是否破坏缓存:不涉及——未触碰任何缓存/去重键/增量游标机制;新增重试只发生在单次摘要调用内部。
  7. 是否改了设计方向:无架构转向;重试裁决复用主链既有分类而非新造体系。一处语义边界调整(按披露要求单列):启动审计不再报告 unknown-provider/unknown-model 引用(boot 时 agent 层 recipe 不可见 → 确定性假阳),这类问题改由 web save 400 + 请求路径一次性 warn 呈现。
  8. 是否引入了新的配置:无——没有新 env/config key/默认值(config-surface-guard CI 通过)。

行为变更披露(旧 → 新 + 原因)

  • 瞬态摘要失败:1 次派发 → 最多 3 次派发(新)。原因:瞬态网络抖动不应直接判死会话;整体仍被 targetTimeoutMs 封顶,最坏墙钟不变。
  • 拒绝类报错文案:...rejected the request → ...rejected the request with HTTP <status>。已验证无消费者依赖旧串。
  • GET /__bili/config 响应:新增 externalSummaryCredentialStatus(纯增量,旧字段不动)。
  • 启动日志:凭证损坏时新增 warn 行;unknown 引用不再产生启动 warn(见第 7 问)。
  • 日志正文片段现在过 URL 脱敏(review 修复,防 URL 内嵌凭据泄漏进日志)。

判定

方向正确、实现闭环、CI 全绿、无回归、无越权配置面变更。可以合并——按规矩合并动作留给你:#3541

@ranxianglei
ranxianglei merged commit 05aae74 into master Oct 18, 2026
48 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

📦 Merged Build Published

This merge to master was published as 0.0.561-master.05aae74 (rolling master tag).

npm install -g billion-context@0.0.561-master.05aae74  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant