Skip to content

fix: cross-spec compress-failure circuit breaker + stale-generation diagnosis (#2146) - #2147

Merged
ranxianglei merged 2 commits into
masterfrom
2026-10-05_compress-loop-circuit-breaker
Oct 6, 2026
Merged

ranxianglei merged 2 commits into
masterfrom
2026-10-05_compress-loop-circuit-breaker

Conversation

@ranxianglei

@ranxianglei ranxianglei commented Oct 5, 2026 •

Copy link
Copy Markdown
Owner

Closes #2146

Problem

After a dsh client restart (plugin mode), the model (glm-5.3-flash) kept calling compress with ranges that could never resolve against the live session — stale-generation refs left in its persisted history. The incident log (bili (17).log, v0.1.179) shows 161 consecutive compress failures over 2h42m, each on a distinct spec (walking spec space: sequential 50-ref windows above the frontier, then tracking slightly ahead of it, finally huge tail ranges), while the model never called acp_status once despite every receipt asking it to. Each round trip burned ~106K input tokens.

Root cause

The existing anti-loop guard is keyed on the EXACT spec (#847: fires only when the same normalized range repeats ≥2×). A model that issues a fresh failing range each call never trips it. The #1026 span hint and #1029 ref snapshot were already inside every failure receipt — present but unread by this model. Nothing tracked "still failing" across DISTINCT specs, so no escalation path existed from diagnostic hints to a hard stop.

Secondary gap: when ALL requested refs sit strictly above the session's highest mapped ref, the kernel's generic "typo or wrong session" wording misleads the model into hunting a typo that cannot exist — the deterministic cause is stale-generation artifacts in the client's persisted history.

Fix (host-side, no kernel change)

src/stream.ts:

  • Cross-spec circuit breaker (noteCompressLoopFailure): a session-level streak of consecutive TOTAL compress failures (any spec), persisted in session.metadata["compressFailStreak"] alongside 出现持续压缩失败,无法自动恢复 #847's map. At 3 consecutive failures the failure receipt swaps the diagnostic hint cluster for a hard-stop paragraph ("STOP calling compress now … Continue your actual task"). Disarms on any successful compress or after 10 min quiet (refs keep advancing DURING the loop, so frontier-based decay would disarm mid-loop). The breaker NEVER refuses execution — it only rewrites what a FAILED call tells the model, so a legitimate call can always still succeed and reset the streak. Warn log on arm and on each armed failure ([warn: compress-loop] …) for operator diagnosis.
  • Deterministic stale-generation diagnosis (beyondFrontierNote): when every requested endpoint is above the session's highest m-ref, the receipt states those refs cannot exist here, names stale artifacts from an earlier generation as the cause, and notes that NO superset of them can ever compress.
  • Wired into all three failure paths of the shared applyRanges (unparseable call / zero-blocks gate+kernel rejection / throw).

src/preflight.ts: preflight overflow-compress passes { loopTracking: false } — bili's own internal lane must not arm the model-facing breaker on attempts the model never made. Its existing startsWith("[Compression FAILED") check is untouched (failure prefix unchanged).

Behavior changes (disclosed)

  1. 3rd+ consecutive compress failure (any spec, within 10 min): receipt was […errors][Current context…][Repeat-failure guard…][span hint][no-viable…], now […errors][beyond-note?] [COMPRESS CIRCUIT BREAKER: N …] — the hint cluster is REPLACED by the hard-stop paragraph (stacking more ignored advice is exactly what failed in the incident). Applies to BOTH modes: wire and plugin share applyRanges.
  2. 出现持续压缩失败,无法自动恢复 #847 repeat guard: unchanged at 2× same-spec; at 3× its text is superseded by the breaker paragraph (one assertion in tests/issue847-compress-failure-guards.test.ts updated accordingly).
  3. All-refs-above-frontier failures: gain one deterministic sentence ahead of the existing hints; first-two-failure receipts otherwise structurally unchanged.

No other output-format, threshold, timing, or default changes.

Config surface

None. Threshold (3) and decay window (10 min) are hardcoded constants by design, per env/config-surface discipline (cf. DEGENERATE_FOLD_COVERAGE).

Both compression modes

Shared applyRanges → plugin mode (dsh, the incident lane via src/loop/core.ts) and wire mode (OpenAI/Responses/Anthropic streams) both get breaker + diagnosis; preflight lane explicitly excluded; success path clears the streak in all lanes.

Tests

  • New tests/issue2146-compress-loop-breaker.test.ts (6): distinct-spec arm at 3 + count continuation + warn logs; success clear; 10-min decay vs 9-min continuation; beyond-frontier receipt text + mixed-endpoint suppression; unit shapes (bN endpoints, empty state); loopTracking:false isolation.
  • One assertion updated in tests/issue847-compress-failure-guards.test.ts (the intended behavior change fix: keep system prompt byte-stable for prefix cache #1 above).
  • Targeted green locally: both files (10/10) plus all 12 receipt-touching test files (89/89); typecheck clean; build clean. Full suite → CI.

Addendum (commit 3773b9f): receipts no longer echo phantom refs back

Owner asked whether the fix was stop-the-bleeding without locating WHY the loop runs. Deeper log analysis found a self-sustaining channel the first commit left open: every failed compress receipt quotes the kernel's per-range error text verbatim (startId="m29971" does not exist…), which lands in the client-persisted history as a tool result — where the looping model reads its own past failures as unfinished work and slides further up the same phantom ladder (incident log: m29971–m30020 → m30021–m30070 → m30071–m30120 within 20s, retries widening to m30121–m30260 hours later).

Fix: when beyondFrontierNote fires (every requested ref provably above the session's highest mapped ref — the stale-generation case), the requested endpoint refs are scrubbed from the model-visible receipt ([stale-ref] placeholder); the server log keeps the real refs for operators. In-frontier unknown endpoints stay verbatim (a genuine typo the model can correct). Safety invariant: when the note fires, no live snapshot number can collide with a scrubbed token. Composes with the #1875 ProxyToolResult envelope (compressResult(receipt, "refused", 0)).

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 收到,开始 review PR #2147(closes #2146)。已拉取分支 2026-10-05_compress-loop-circuit-breaker(7e7aa6f,diff:src/stream.ts +112、src/preflight.ts +3/-1、新测试 151 行、issue847 断言更新 1 处)。CI 尚处 pending;分支基点(5e43209)落后当前 master(#2139/#2131 等已合入),正在核对 rebase 冲突与既有行为回归。完整结构化结论稍后补在同线程。

@github-actions

github-actions Bot commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-05_compress-loop-circuit-breaker (62bb0c7)

Option A — Install this exact build (recommended)

npm install -g billion-context@0.0.2381-pr.2147.fb3dff0

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr2147.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 Review 完成。分支原本与 master 冲突(mergeable_state=dirty),已 rebase 到最新 master 并解决冲突后推送:984c8d4(原 7e7aa6f)。CI 20/20 全绿,PR 现 mergeable: true / state: clean。

本轮 review 做的事

结构化七问

  1. 是什么问题:客户端重启后持久化历史里残留上一代 session 的 ref,模型反复拿这些必然失败的 ref 调 compress,且每次都换一个"新"spec(出现持续压缩失败,无法自动恢复 #847 的同-spec 重复守卫永远不触发),事故日志 2h42m × 161 次不同 spec 失败,每轮烧 ~106K input tokens。
  2. 什么原因导致的:现有防循环机制按「完全相同 spec ≥2 次」计数——模型走 spec 空间(每次一个新范围)就绕过了它;receipt 里的 失败 compress 回执中给出当前可压缩区间 — 社区 PR #1008 (@asuan-dev) #1026/ACP 状态标记被拼接进用户消息文本块;跨会话代际 stale refs 压缩整批失败 #1029 提示模型从未读取;没有任何机制跟踪「跨 spec 的连续总失败」。次要缺陷:全部 ref 都高于 session frontier 时,kernel 的通用 "typo or wrong session" 措辞把模型引向找错别字而非识别 stale-generation 根因。
  3. 怎么修复的:host 侧两道机制,均落在共享 applyRanges 的全部三条失败路径(unparseable / zero-blocks / throw):① 会话级连续总失败 streak(persisted session.metadata["compressFailStreak"]),第 3 次起 receipt 的诊断提示簇整体换成硬停段("STOP calling compress…"),成功或 10 分钟静默解除,断路器从不拒绝执行(合法调用仍可成功并重置);② beyond-frontier 确定性诊断句(所有端点严格高于最高 m-ref 时点名 stale-generation 根因)。preflight 内部 lane 传 {loopTracking:false} 隔离。
  4. 是否有回归问题:无。逐项过触碰路径的既有行为——Plugin tool endpoint reports ok:true when the kernel refuses a fold range #1875 outcome 契约(applied/partial/refused+blocksCreated):未动,issue1875 套件绿;出现持续压缩失败,无法自动恢复 #847 守卫状态机(compressFailKeys 映射、成功清除):逻辑未动,仅 armed 时展示被替换(已披露);失败 compress 回执中给出当前可压缩区间 — 社区 PR #1008 (@asuan-dev) #1026 span hint / ACP 状态标记被拼接进用户消息文本块;跨会话代际 stale refs 压缩整批失败 #1029 snapshot / [Bug] Codex 新会话首轮无可压缩内容仍调用 compress,低于 5000 字符阈值后进入 ACP 工具循环 #1112 no-viable / compress failed may be caused by params #1494 dropped-entries / 反向 ref 注记:not armed 时全部保留,armed 时 kernel 错误文本与 dropped 注记仍追加;preflight overflow-compress 的 noteSkip/skipSet 流:未动(仍 keyed on outcome==="refused");持久化:仅新增自由 metadata key(与 出现持续压缩失败,无法自动恢复 #847 同纪律),旧进程忽略未知 key,无格式版本变更;wire 形状/工具调用顺序/SSE:未动,两种压缩模式都推演过(plugin mode src/loop/core.ts:282 与 wire lanes 默认 tracked,preflight 排除);nudge 节奏/缓存账本/session identity/kernel:未触碰(kernel-guard 绿)。命名核查:新 [warn: compress-loop] 日志标记与既有 src/compress-loop-responses.ts(Responses wire 的 MAX_LOOP_ROUNDS,日志措辞不同)无 grep 冲突。
    行为变更(均为 PR 已披露项,逐条核实):① 第 3 次连续失败(任意 spec、距上次 ≤10min)receipt 提示簇被硬停段替换,双模式生效;② 出现持续压缩失败,无法自动恢复 #847 2× 文本不变,3× 起被断路器段取代(对应断言已更新);③ 全 ref 越界的首次失败即多一句确定性诊断(加法式)。
  5. 是否破坏缓存:不涉及。前缀缓存机制未动;断路器文本只出现在失败调用的 receipt 里,稳定 system 前缀字节不变。
  6. 是否改了设计方向:没有。host 侧诊断增强,留在既有失败路径内,kernel 未动,无架构转向。修法针对的是产生缺陷的机制(无界的跨 spec 失败循环)而非掩盖症状:不拒绝执行、不过滤字节、只在循环形态成立时改变失败回执内容,并在可证明时点名确定性根因(stale refs 本身产自客户端跨代持久化,超出 bili 边界,处理正确)。
  7. 是否引入了新的配置:无。阈值 3 + 10 分钟衰减按设计硬编码(先例:DEGENERATE_FOLD_COVERAGE),无新 env/config 字段。

测试有效性:6 个新测试走真实 kernel 执行(applyRanges × 真实 session state),断言钉死 receipt 精确措辞与 metadata 状态(arm@3、continue@4、warn 日志、成功清除后重启、11min 衰减 vs 9min 延续、beyond-note 文本+混合端点抑制、bN 端点/空态单元形、loopTracking:false 五项隔离)——有牙,能抓住该机制的回归;issue847 更新断言钉住 3× 时的取代关系。已知缺口:没有端到端复现完整事故场景(stale-generation 重启)的 hermetic harness,单测钉桩 + CI 真实客户端 lane 覆盖下可接受。

判定:可合并(mergeable clean,CI 20/20 绿)。合并动作留给人:#2147

@ranxianglei
ranxianglei force-pushed the 2026-10-05_compress-loop-circuit-breaker branch from 3773b9f to 40097a2 Compare October 5, 2026 08:39
@ranxianglei
ranxianglei force-pushed the 2026-10-05_compress-loop-circuit-breaker branch from 40097a2 to 62bb0c7 Compare October 6, 2026 15:11
@ranxianglei

Copy link
Copy Markdown
Owner Author

已 rebase 到当前 master(818d24e+)——零冲突,全绿。 本地验证:tsc 干净、npm run build 过、PR 自带 issue2146-compress-loop-breaker + issue847 守卫回归、preflight 家族(#1933 校准等)25/25、cache-ledger/stream 37/37。

与 #2110 家族的关系:本 PR 修的是"压缩一直失败无降级"(161 连败烧 106K×N input 的陈旧 generation 场景),与 #2110 的"无人压缩"互补——失败侧的止损。熔断逻辑不掩盖任何东西:每次失败照常入账,跨 spec 累计才升级为 hard stop + stale-generation 诊断,与 #2110 留言里"失效要显性失败"的处置一致。

可独立 review 合并;与 #1818/#1851 同在 preflight/compress 区域但不冲突(本 PR 主要在 src/stream.ts)。

@ranxianglei
ranxianglei merged commit 9f82bbe into master Oct 6, 2026
23 checks passed
@github-actions

github-actions Bot commented Oct 6, 2026

Copy link
Copy Markdown
Contributor

📦 Merged Build Published

This merge to master was published as 0.0.88-master.9f82bbe (rolling master tag).

npm install -g billion-context@0.0.88-master.9f82bbe  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

dsh内部问题 出现这个错误后,重启dsh可以发消息 但陷入无限压缩失败

1 participant