Skip to content

fix: preflight classifies normal-stop zero-text summaries as deterministic (#2832) - #2840

Merged
ranxianglei merged 1 commit into
masterfrom
2026-10-12_preflight-deterministic-empty
Oct 12, 2026
Merged

ranxianglei merged 1 commit into
masterfrom
2026-10-12_preflight-deterministic-empty

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

Closes #2832

What was wrong

A relay model that answers every preflight summary probe with a NORMAL stop (end_turn on the anthropic wire, stop on openai, completed on responses) and zero answer text — the whole answer riding in thinking blocks, or the relay stripping the text blocks — was classified transient by preflight (no size signal matched). Every span therefore burned two same-span retries plus the per-regime retry budget, then the #726 halving cascade re-billed doomed sub-spans, for EVERY span of EVERY range EVERY turn. Incident (2026-10-11, bili 0.1.188): six failed probes over six minutes, ZERO folds landed, no sticky verdict, no actionable advice — the turn fell through to an estimate-only evidence forward that billed the full payload. The diagnosis string also could not tell thinking-only from relay-stripped-text, so operators had nothing to act on.

The fix (mechanism-level)

  1. Classification. New exported isDeterministicEmptyCompletion(protocol, json): a plain-JSON completion that ended normally yet carries zero extracted answer text is deterministic for that (model, host) pair — re-sending the identical payload returns the identical empty answer. Such outcomes set transient: false (no bili会突然暂停发继续又可用 #1767 same-span retries) and a new deterministic flag. Non-zero short answers, SSE bodies, size-driven stops, content_filter, truncated/incomplete streams all stay exactly where they were.
  2. No halving for deterministic empties. A normal stop is not a size signal, so the [Bug] Codex 额度中断后恢复长会话,preflight SSE 摘要为空并持续 502 #726 halving cascade is skipped: the range goes straight to the give-up path (one billed call instead of draw + retries + halving cascade).
  3. Sticky dead-probe verdict. Three consecutive deterministic empties on the same (host, model) key arm a verdict persisted in session metadata (deterministicEmptyProbe, same discipline as the existing learned flags). While dead, at most ONE canary probe per preflight invocation bills the wall; the walk stops with an actionable fail-fast detail. A successful canary clears the verdict and folding resumes on a self-healed relay. Keyed by (host, model), so a stale entry is inert after a relay/model switch; the external-summary lane bypasses the verdict entirely (different executor).
  4. Diagnosis fingerprint. emptyCompletionDetail() now reports where the model's output went: text_chars=N / text_blocks=N / thinking_blocks=N / reasoning_chars=N (openai + anthropic) and text_parts=N (+ reasoning_chars) on responses — logs and fail-fast details finally distinguish thinking-only from stripped-text from truly-nothing.

Behavior changes (old → new)

Surface Old New
end_turn/stop/completed + 0-text probe transient → 2 same-span retries + shared retry budget + #726 halving cascade, every span/range/turn deterministic → no retries, no halving, one strike recorded
Repeated deterministic empties unbounded re-billing across spans/ranges/turns 3 strikes arm a session-persisted verdict; ≤1 canary probe/invocation while dead; cleared by any successful model-path probe
Diagnosis string (finish_reason=stop, answered as model=X) gains block fingerprint, e.g. (stop_reason=end_turn, text_blocks=0, thinking_blocks=2, reasoning_chars=1873, answered as model=k3-256k) — three pinned test strings updated accordingly

Size-driven (#726/#853/#874), transient (#1767/#780/#784), external-summary, batch-fold, futility (#1841), round-budget (#574/#1933), and dead-end-cooldown behavior are unchanged.

Both compression modes

Preflight fires in BOTH modes as the last-resort backstop when the input alone exceeds the window (TECHNICAL-NOTES.md "Two compression modes", #470). This change touches only probe classification, retry/halving gating, and walk stopping — no summary carrier, no system-message handling, no tool/nudge injection — so plugin mode (agent-owned compression) and proxy mode get identical treatment. Verified: sole entry is preflightCompressIfNeeded (src/server.ts:2354) called from src/server/handle.ts:2011; no carrier/rendering paths touched.

Config surface

None added. The strike limit is a hard constant (DETERMINISTIC_EMPTY_STICKY_LIMIT = 3, matching the auto-update 3-strikes convention); the verdict lives in internal session metadata, not config/env/CLI. The operator exits already exist and are named in the advice: compress.externalSummary, switching the session model, or fixing the relay.

Tests

  • New tests/issue2832-preflight-deterministic-empty.test.ts: classifier matrix (14 cases incl. thinking-only→deterministic, any-text→not, max_tokens/length/content_filter/incomplete→not, google/SSE/null→not), exact fingerprint strings per wire, systemic e2e (verdict arms within 3 turns, total strikes bounded 3–6 vs unbounded before, nothing forwarded, EXACTLY one canary call on the carried-verdict turn, verdict survives), recovery e2e (canary clears → fold proceeds → 200, metadata deleted).
  • Existing pins updated where the fingerprint legitimately extends them (preflight-empty-summary, preflight-transient-empty-retry).
  • tests/preflight-multi-range.test.ts fixture moved from end_turn+empty to max_tokens+empty ON PURPOSE: its failure shape was silently inside the class this PR reclassifies, which would let the dead-probe verdict shadow the budget-cap mechanism that file exists to pin. It now stays on the size-driven halving path its "2 calls per failed range" arithmetic describes (provenance comment added).
  • Local targeted runs green: new suite 4/4; full preflight/summary/fold domain 517 pass / 0 fail (all files matching preflight|summary|fold|external-summary|compress-reject etc.); typecheck clean. Full suite + e2e lanes to CI.

Kernel untouched; pi-subagents untouched; no version bump.

…istic (#2832)

- diagnoseEmptySummary gains per-wire block fingerprints (text_chars / text_blocks / thinking_blocks / reasoning_chars / text_parts) so a thinking-only answer is distinguishable from relay-stripped text in logs and fail-fast details
- new isDeterministicEmptyCompletion: a normal stop (end_turn / stop / completed) carrying zero answer text is deterministic for that (model, host) pair — it earns neither #1767's same-span retries nor #726's halving recovery, because re-sending the identical payload returns the identical empty answer and a normal stop is not a size signal
- three consecutive deterministic empties on the same (host, model) arm a session-persisted dead-probe verdict (session.metadata deterministicEmptyProbe); while dead, at most ONE canary probe per preflight invocation bills the wall, and a successful canary clears the verdict so folding resumes on a self-healed relay
- fail-fast detail names the actionable exits: configure compress.externalSummary, switch the session's model, or fix the relay's thinking/text handling
- tests: classifier matrix + fingerprint pins (unit), systemic e2e (bounded strikes, sticky verdict, exactly-one-canary contract, nothing forwarded), recovery e2e (canary clears, fold proceeds, 200); #574 multi-range fixture moves to max_tokens so that file keeps pinning the walk/budget mechanics on the size-driven path independent of this reclassification
@github-actions

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-12_preflight-deterministic-empty (f7a247f)

Option A — Install this exact build (recommended)

npm install -g billion-context@0.0.3143-pr.2840.e2f862d

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr2840.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 PR #2840 复核完成。本 PR 是我上一轮针对 #2832 的实现;本轮按 [PR] 唤醒规则拉取分支核对代码、确认 CI 并给出审查结论(未重做)。

核查范围:单 commit f7a247f,基于 master cd02d2d(线性、无落后),5 文件(src/preflight.ts +217、新测试 +323、3 个既有 pin 更新)。逐段读了 src/preflight.ts 实际代码(非仅 diff hunk)+ 全部测试。

① 是什么问题:preflight 把「正常结束(end_turn/stop/completed)+零正文」的确定性空摘要误判为瞬时故障——每个 span 烧 2 次同-span 重试 + 共享预算 + #726 减半级联,跨 span/range/turn 无上限计费,且诊断串无法区分「全答在 thinking」与「中继吞掉 text」。

② 什么原因导致的:requestSummary 短分支用 transient = !emptySummaryIsSizeDriven(diagnosis) 二分——没有尺寸信号一律当瞬时。end_turn+0 正文既非瞬时光(会自愈的抖动)也非尺寸驱动(减半无用),是第三类 deterministic。旧代码无此类、也无跨 span/turn 记忆。修法打在分类源头(producer),不是中途过滤症状——符合「疏不堵」。

③ 影响范围与严重程度:走 preflight LLM fold 且摘要模型经中继回 thinking-only 正常结束(或中继剥 text)的用户;触发 = 超窗 + 该形态稳定复现。特定条件下完全无法折叠(每 turn 烧光探测预算、零 fold),最终 fail-fast 502 或落入 #2313 evidence-forward 全额计费空 turn(即事故)。非该形态用户不受影响。

④ 怎么修复的(机制层三件事,已逐一读码验证):

  1. isDeterministicEmptyCompletion(纯函数):正常停止 + 零抽取文本 → deterministic;排除 length/max_tokens/content_filter/incomplete/SSE(null)/非对象/任何非零文本。→ transient:false,不进 bili会突然暂停发继续又可用 #1767 同-span 重试 while(条件含 part.transient,已核实 :2145)。
  2. 不走 [Bug] Codex 额度中断后恢复长会话,preflight SSE 摘要为空并持续 502 #726 减半(:2275 deterministicUnusable → noteSkip+skipSet+break,先于 halving cascade)。
  3. sticky verdict:{key,strikes,dead} 存 session.metadata["deterministicEmptyProbe"](同 Codex(ChatGPT 登录)preflight 摘要被 400「Stream must be set to true」拒绝,长会话返回 502 #626 learned-flag 纪律),key=host|model 换模型即失效;externalSummary 通道整体绕过(probeGated=false)。边界已核实:fresh dead relay 每 range ≤1 strike、第 3 strike arm、range 门(:1934)停整 walk;carried verdict 每 turn 恰好 1 canary(:2142 调用前预置 canarySpent);canary 成功 clearDeterministicProbe(:2166)恢复折叠;strike 跨 turn 累计(持久化后读回)。

⑤ 是否有回归问题(逐项清点既有行为):尺寸驱动(#726/#853/#874 max_tokens/length 仍减半)✓保持;瞬时光(#1767 content_filter、#780/#784 truncated/incomplete 仍重试)✓保持;external-summary / batch-fold / futility(#1841) / round-budget(#574/#1933) / dead-end cooldown ✓均未触碰。一处测试夹具配套改:preflight-multi-range.test.ts 失败形态 end_turn+空 → max_tokens+空(带 provenance 注释)——其旧形状恰落在本 PR 重分类的类里,不改会让新判决遮蔽该文件要钉的预算机制;生产语义零变化。结论:无意回归 0 处。

⑥ 是否破坏缓存:不涉及。不碰消息重建/system 锚点/prefix-cache 字节;diff 只落 src/preflight.ts(+tests),metadata 新增字段是内部状态,不影响客户端历史与 wire。

⑦ 是否改了设计方向:不涉及架构转向。在既有 transient/size-driven 二分上补第三类 deterministic,判决持久化沿用 session.metadata 纪律。两种压缩模式行为一致(preflight 本就是双模式 last-resort backstop #470;改动只涉分类/retry/halving/walk-stop,不碰 carrier/rendering——diff 未触及 server.ts/adapter)。

⑧ 是否引入新配置:无。阈值硬常量 DETERMINISTIC_EMPTY_STICKY_LIMIT=3;判决存内部 metadata,不加 env/config/CLI。operator 出口用现有表面(compress.externalSummary/换模型/修中继),已写进 arm 时 warn 日志与 fail-fast 详情。

测试有效性:真测试有牙。classifier 矩阵 14 例钉住每个边界(content_filter/max_tokens/length/incomplete/google/SSE 均判 not-deterministic);指纹串精确匹配区分 thinking-only vs stripped-text;两个 e2e 断言 bounded billing(3–6 vs 旧 unbounded)+ exactly-one-canary + self-heal 清 verdict。3 个既有 pin 更新都加了牙(text_chars/text_blocks 精确串)+ content_filter 仍 transient 断言。

CI:Linux 双 node(22/24)全绿;guard 全绿(lane/docs/lock/version/config-surface/kernel-guard/build/image-billing);Windows 双 node 仍在跑(同一套套件,Linux 已过)。

收工判定:代码核对成立、方向正确(修根因非症状)、无意回归 0、测试有牙、Linux CI 全绿。等 Windows CI 收尾后即可由人合并 PR #2840,我这边无需再改动。

@ranxianglei
ranxianglei merged commit b7be571 into master Oct 12, 2026
15 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

📦 Merged Build Published

This merge to master was published as 0.0.297-master.b7be571 (rolling master tag).

npm install -g billion-context@0.0.297-master.b7be571  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant