Skip to content

[PR] fix: external-summary failures now carry classified diagnostics, references fit each candidate's real window, and tier-2 nudges stop self-amplifying on persistent failure - #3659

Open
ranxianglei wants to merge 6 commits into
masterfrom
2026-10-18_external-summary-failure-diagnostics
Open

ranxianglei wants to merge 6 commits into
masterfrom
2026-10-18_external-summary-failure-diagnostics

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

Closes #3638

查重: 无(平台 issue 检索无同症状条目;与已合入的 #3541/#3488 相关但机制不同——它做运维侧重试+日志,本 PR 做调用方可见诊断、reference 窗口适配、持久失败期的 nudge 抑制;分支已 rebase 到含 #3541 的 master,唯一冲突按并集解决)。

What was broken (verified against v0.1.199 == reporter's build)

Every T2/T3 distillation in external summary mode failed with one generic sentence, for ANY span size, while each turn re-injected the full emergency tier-2 block catalog. Three coupled defects, one failure domain:

  1. Universal overflow: every work item's reference payload carried ALL other active T1 summaries + uncovered context UNBOUNDED (~286K tokens in the reporter's session). The pre-send window check counted that unbounded reference against the candidate's window, so once accumulated summary mass outgrew the window, even a two-block span overflowed identically.
  2. Opaque diagnostics: the executor swallowed per-attempt error text into {outcome:"error"}; the tool result was one sentence for config-missing / credential-fail / provider-reject / input-overflow / output-overflow / timeout-exhaustion alike.
  3. Self-amplification: the kernel tier>=2 nudge fires on token mass alone and renders the UNCAPPED block catalog every turn; preflight folds (no reference) kept succeeding and adding T1 mass while the only authorized distillation path was dead — no escape hatch.

The fix (host-side only, no kernel changes)

A. Classified bounded secret-free diagnostics. Every failed attempt carries failure:{cls,detail?} with cls in {candidate_unavailable, input_budget_overflow, provider_rejected, network_error, timeout, invalid_output, output_budget_overflow, unknown_error}. The tool result becomes per-candidate lines <name> (<protocol>/<model>): <cls> — <detail> (last attempt per targetIndex, max 3 lines) plus congestion/deadline notes and a safe-recovery tail, hard-capped at 900 chars. Security contract preserved and extended: response bodies never surfaced (status code only), URLs scrubbed, credential values excluded from the chain fingerprint, raw provider text never enters results (existing without leaking provider errors pins honored).

B. Reference fitted to each candidate's real window. Work items now carry structured referenceParts; before sending, system+content+reference+output are accounted against contextWindow, and if it doesn't fit the OLDEST otherSummaries are dropped first, then oldest context messages (block order verified oldest-first). Still overflowing → typed input_budget_overflow with token numbers and ZERO requests sent. Trim is logged info-level. Tiny spans stop carrying the whole session's summary mass — this is the availability fix for the reported symptom.

C. Persistent-failure nudge dampening. Per-session latch in session.metadata: after 2 consecutive all-candidate-failed batches under an unchanged chain fingerprint, the tier>=2 distillation/condensation nudge (EMERGENCY header + full block catalog) is replaced by a 3-line standing notice (unavailable status + last failure classes + recovery path: fix config or set compress.externalSummary.enabled=false). First successful fold or any config change clears/resets it. Congestion outcomes (#2662 pool-saturated, zero dispatches) neither arm nor clear. Swapped ONLY in the final-else render branch of all four lanes (anthropic/openai/responses/google); the decide-fallback tier===1 branch and every non-external-mode session stay byte-identical.

Behavior changes (old -> new, intentional)

# Change Old New
1 Tool-result failure message one generic sentence classified per-candidate diagnostic (class + bounded detail, <=900 chars, secret-free) + safe recovery guidance
2 Reference payload unbounded (all other summaries + context) fitted to each candidate's real window, drop-oldest shrinkage (summaries may differ subtly across window sizes — intended)
3 Tier>=2 nudge while external chain fails persistently full block catalog re-injected every turn concise standing failure notice after 2 consecutive all-failed batches (same config); auto-resumes on first successful fold or config change
4 Operator logs batch status line per-attempt classes, reference trim counts

No new config keys / env vars / CLI flags; no kernel/ changes; no version bump.

Coordination with #3541 (merged to master during this PR's life)

Rebased onto current master. Single conflict in src/external-summary-http.ts resolved by union: #3541's bounded retry loop stays intact; its final non-retryable rejection now throws the typed provider_rejected error (status surfaced, body still log-only). Its exhausted-chain test pin extended with the classified failure field (contract extension, labeled).

Verification

Local targeted suites on the rebased head: external-summary/nudge family 204 pass, preflight family 185 pass, compress/lane-rewrite family 203 pass, 0 fail; typecheck clean. Full-suite verdict = CI on this head.

(ai: vllm-qwen/qwen3.8-27b)

@ranxianglei ranxianglei changed the title fix: external-summary failures now carry classified diagnostics, references fit each candidate's real window, and tier-2 nudges stop self-amplifying on persistent failure [PR] fix: external-summary failures now carry classified diagnostics, references fit each candidate's real window, and tier-2 nudges stop self-amplifying on persistent failure Oct 18, 2026
@github-actions

github-actions Bot commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-18_external-summary-failure-diagnostics (51d21c4)

Option A — Install this exact build (recommended)

npm install -g billion-context@0.0.4447-pr.3659.709d472

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr3659.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重: 不重复。 按失败签名(external-summary 全候选失败的泛化文案 + tier-2 目录 nudge 自放大)在 open+closed issue/PR 检索:无同症状条目。相关但机制不同:已合入的 #3541(修 #3488)做的是运维侧重试循环 + per-attempt 日志 + 凭据三态 UI;本 PR 做的是调用方可见的分类诊断、reference 按候选真实窗口适配、持久失败期的 nudge 抑制——三个机制均不在 #3541 覆盖范围内。两者在 src/external-summary-http.ts 的交集(rebase 冲突点)我会在审查时单独核对并集是否正确。

开始独立审查:拉分支读全量 diff、清点触碰路径的既有行为、核对测试有效性。CI 状态一并确认。结论稍后按结构化八问给出。

awork added 2 commits October 18, 2026 15:30
…print on every side (#3638 review)

renderLaneNudgeText passed the full request Config to
externalSummaryFingerprint, which expects the chain object and returns
"off" for anything else — so the tier>=2 standing-notice swap could
never arm in any lane (the self-amplification fix was dead code). The
plan also fingerprinted its PARSED settings while the catch paths and
lanes see the raw rail chain; parseExternalSummarySettings fills
contextWindow/outputTokens defaults and normalizes URLs, so the two
fingerprints disagree whenever a target omits those fields.

All three sides now fingerprint the raw rail chain: the plan records
fingerprint(raw), the lanes compare fingerprint(config.externalSummary),
and the catch paths already used it. New lane-seam regression test pins
the full-Config argument shape, the plan/raw parity, and the negative
rows (tier-1 untouched, mode-off untouched).
…tion test (#3638 review)

EXTERNAL_FAIL_LATCH_THRESHOLD is only used inside
external-summary-failures.ts — drop the leading export so the
unused-exports gate passes again. The provider-rejection test slept
real backoff (1s + 2s) inside a 4s total budget, so on slow machines
the third attempt hit the deadline and classified as timeout instead of
provider_rejected (deterministic failure here, failed 3/3); zero the
backoff through the existing #3488 test seam like the other retry
tests do.
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

审核结论(独立审读全量 diff,14 文件 +629/-45):方向正确——三缺陷同属一个失败域(host 侧修复、不碰 kernel)。但审核发现并直接修掉 1 个真实缺陷:特性 C(持续失败时替换 tier≥2 nudge)在合入形态下是死代码,latch 永远无法武装;另修 1 处 CI 红灯和 1 个依赖机器速度的测试。两个标注 commit 已推到 PR 分支(c028d16a1、db5cc1b91)。

审核发现并已修(直接修,不等反馈循环)

  1. latch 永不武装(死代码):renderLaneNudgeText 把整个请求 Config 传给 externalSummaryFingerprint(四个 lane 调用点都传完整 config,如 src/prepare-anthropic.ts:467),而该函数期望 chain 对象、对其它一律返回 "off"(src/external-summary-failures.ts:131);记录的指纹又永远不会是 "off"(只在 enabled=true 时记录)→ 相等比较恒假 → tier≥2 standing notice 在任何 lane 都不会触发。新增测试没覆盖这个 seam——它们直接用匹配字符串调 externalFailureLatchActive,绕过了 lane 入口。修法:三个触点统一对 raw rail chain 取指纹——plan 记录 fingerprint(raw)(src/external-summary-runtime.ts:44)、lane 比对 fingerprint(config.externalSummary)(src/nudge-decide.ts)、catch 路径本来就是。补了 lane-seam 回归测试(full-Config 参数形状 + plan/raw 一致性断言 + tier-1/mode-off 负向行)。
    • 细节:只改 lane 侧传 .externalSummary 还不够——parseExternalSummarySettings 会填 contextWindow/outputTokens 默认值(128K / min(8K, 窗口/4))并规范化 URL(src/external-summary-settings.ts:287-295),用户省略这些字段时 parsed 与 raw 指纹不同。所以 plan 侧也必须一起改。测试 fixture 故意省略窗口字段钉住这个 case。
  2. CI 红灯:unused-exports guard 失败——EXTERNAL_FAIL_LATCH_THRESHOLD 导出但只在模块内使用 → 去掉 export(本地先复现后修)。
  3. 测试依赖机器速度:429 分类测试真实 sleep 了 1s+2s 退避,吃掉 4s 总预算的大部分,慢机器上第三次尝试被判 timeout 而非 provider_rejected(本环境确定性失败 3/3;干净 PR head 上同样失败——而 CI 里所有 test lane 都是 skipped,所以没暴露)。按 External summary credential feature entirely unusable + fully opaque failures + no retry | 外部摘要(externalSummary)凭据功能整体不可用 + 失败全程黑箱 + 无重试 #3488 既有惯例用 _setSummaryRetryDelayForTest(0) seam 归零退避。

触碰路径的既有行为回归清点(逐项)

  • fix: external summary credentials — readable failures, bounded retry, tri-state status (#3488) #3541 重试环:原样保留(SUMMARY_MAX_ATTEMPTS=3、Retry-After cap、fail-fast 分类、masked 日志);唯一变化是最终非可重试拒绝从字符串 Error 改为 typed ExternalSummaryError(provider_rejected,status 透出、响应体仍只进日志)——有意为之,分类所需。
  • preflight:work items 不带 reference/referenceParts(两个生产点 src/preflight.ts:1131 单条 / :2239 批量分片只有 instructions/content/尺寸约束)→ 无界引用缺陷只存在于 model-tool 路径,preflight 行为不受影响。
  • executor 兼容:outcome 枚举值不变(success/error/timeout/invalid_summary/cancelled);attempts 加 additive failure?;cancelled 不带 failure(不算链故障)。
  • fitsWindow:与旧 pre-send 检查语义一致(window 或 output 缺失即跳过;sys+payload+256+output ≤ window);drop-oldest 顺序 otherSummaries 先于 currentContext、最新条目存活(真 HTTP upstream 测试钉住 summary-29- 保留 / summary-0- 丢弃)。
  • latch 语义:阈值 2 连续批次;成功清除;congestion-only(auto-fold backoff arms on pure queue-drops (attempts=[]) as if the summary service had failed #2662,dispatched=false)既不武装也不清除;config 变更重置 streak;classes 并集上限 4。
  • 非 external 模式会话:mode 关闭时 fall through 到 renderNudgeText,字节级不变;tier-1 不动;decide-fallback tier===1 分支不动。
  • 安全面:classifyError 不回显任何原始抛出文本(每类固定 detail 串);sanitizeDetail 洗 URL/控制字符、上限 160 字符;fingerprint 排除凭据值(仅存在性 ref/inline);合成消息硬上限 900 字符;"no leaked provider errors" pin 保留并扩展。
  • 既有测试 pin:扩展了 classified 字段(如 tests/external-summary-http.test.ts 耗尽链 pin 增加 failure:{cls:"provider_rejected",detail:"HTTP 503"})——标注过的契约扩展;旧错误串仅被 classifyError legacy 正则消费,无其它消费者。

小问题(不拦合并,建议作者知悉)

  • PR 正文说 per-candidate 诊断「max 3 lines」,代码实际只有 900 字符总上限(无逐行数限制)——有界性成立,措辞与实现不符。
  • 未披露的 wire nuance:model-tool 路径的 reference 由 JSON string 变为 JSON object——用户配置的 summarizer 端点收到的请求形状有细微变化,建议在 PR 行为变更表补一行。

有意行为变更(PR 已声明,核实成立):通用一句话失败 → per-candidate 分类诊断(≤900 字符)+恢复指引;无界 reference → 按候选窗口 fit(drop-oldest 收缩,摘要在不同窗口下可能略有差异——设计意图);持续失败(同配置连续 2 批全败)后 tier≥2 nudge 全文目录 → 简洁 standing notice,首次成功折叠自动恢复;operator 日志 batch 行 → per-attempt class + trim 计数。无新配置/env、无 kernel 变更、无版本 bump。

结构化八问

  1. 是什么问题:external-summary 模式候选链整体失败时,诊断是一句通用话、发给候选的 reference 可能超出其自身窗口、且失败期间 tier≥2 蒸馏目录每轮继续注入自我放大,会话没有出口。
  2. 什么原因导致的:①reference 打包全部其它活跃摘要且无界,pre-send 预算检查却拿候选的小窗口比 → 必然溢出;②错误以未分类字符串抛出,边界吞成通用文案;③nudge 渲染不感知失败状态。审核另发现接线缺陷:latch 比较两侧用了不同对象(整 Config vs chain),恒不相等。
  3. 影响范围与严重程度:启用 external-summary 且链坏掉的用户(配错/限流/宕机)——压缩完全停摆 + 上下文每轮自增,体验毁灭级(p1);正常链路用户零影响。
  4. 怎么修复的:每次尝试分类(8 类 taxonomy,secret-free,有界);reference 按候选真实窗口 fit(drop-oldest,溢出零发送);per-session sticky latch 把 tier≥2 目录换成 standing notice(成功/换配置自动复位);审核修复统一三处指纹源为 raw rail chain。
  5. 是否有回归问题:逐项清点如上——无回归;两处审核修复本身是纯修复(死代码接线 + 测试确定性 + 无用导出),无语义漂移。
  6. 是否破坏缓存:否。不动前缀缓存/稳定 system/prompt pack;standing notice 只替换消息尾部的 nudge 文本,位于前缀缓存断点之后。
  7. 是否改了设计方向:否。host 侧,kernel 不变;latch 沿用既有 session.metadata 纪律(同 autoFoldBackoffUntil/nudgeDecideFails 模式)。
  8. 是否引入新配置:否。恢复路径复用既有 compress.externalSummary.enabled=false。

CI 状态:推送前该 head 上所有 test/e2e lane 为 skipped、unused-exports 红;推送两 commit 后 guard 在新 head 重跑(unused-exports 本地已验证绿),全套判定等 CI。

判定:CI 绿后可合。核心两类修复(分类诊断 + reference fit)成立;latch 特性原为死代码,经本次接线修复才真正生效——#3638 的自我放大症状要到这两个 commit 之后才算真正堵住。合并仍由人来操作。

awork added 2 commits October 19, 2026 10:12
ae1694b was committed from a stale pre-review worktree and silently reverted c028d16 (rail-chain fingerprint wiring in runtime/nudge-decide + lane-seam test) and part of db5cc1b (retry-backoff de-flake of the 429 test). This commit restores those files byte-for-byte to the db5cc1b state; the latch-threshold un-export they contained is preserved. Verified: unused-exports gate clean, typecheck clean, external-summary+nudge targeted suites green.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

External summary mode: every T2 distillation attempt fails, while emergency nudges keep accumulating

1 participant