Skip to content

fix(preflight): log+dump rejected summary bodies and shrink-retry hard 400/413 - #1994

Merged
ranxianglei merged 2 commits into
masterfrom
2026-10-03_preflight-400-recovery
Oct 3, 2026
Merged

ranxianglei merged 2 commits into
masterfrom
2026-10-03_preflight-400-recovery

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

What

#1993: a preflight summarization call answered HTTP 400 left the session wedged and undiagnosable — the response body was never logged or dumped, there was no chunk-shrink fallback (unlike the unusable-HTTP-200 path), and the turn fail-fasted 502 retryable=false plus armed a dead-end cooldown.

Changes

  1. Surface the rejection body — a bounded 200-char snippet (safePrefix) goes into both the warn log line and the client-visible fail-fast detail, parity with the main request paths (src/loop/core.ts, src/server.ts).
  2. Dump the summary exchange — new dumpSummaryRejection(status, sessionId, request, response) in src/error-dump.ts under the same BILI_DUMP_4XX=1 / BILI_DUMP_4XX_MAX_BYTES gate as dumpRejectedBody. Writes summary-err-<ts>-<sid>-<status>.json: a JSON envelope {status, request, response} with each side embedded as a string so the envelope stays valid JSON even when a side is not itself JSON. The distinct summary-err- prefix keeps a main-path and a summary dump landing in the same millisecond from clobbering each other. Called from requestSummaryBody at final throw for 4xx responses — the only site where both byte streams are in hand.
  3. Shrink-retry hard 400/413 — a non-transient 400/413 on a splittable span now gets the same halving cascade as an unusable HTTP-200 summary (halves pushed onto the span worklist; bounded by MAX_SUMMARY_CALLS_PER_PREFLIGHT via the existing while-top budget check). Transient 4xx (risk-control markers) keep replay-retry semantics; other 4xx (auth/routing) fail fast without burning the budget; unsplittable spans give up exactly as before.
  4. Clear stale failure after a successful fold — a fold that succeeds after an earlier rejection clears the recorded failure, so a later exhaustion reports its own reason and a fitting result reports none at all.

Behavior changes (old → new)

# Surface Old New Why
1 warn log + fail-fast message on summary 4xx generic rejected by the upstream (HTTP 400) appends : <first 200 chars of the rejection body> operators need WHY, not just that
2 BILI_DUMP_4XX=1 artifacts forwarded-body dumps only (err-*.json) also rejected summary exchanges (summary-err-*.json) the summary call is built independently of the wire body, so the main-path dump sites never see it
3 hard 400/413 on a splittable span instant fail-fast after one call halving cascade up to the 16-call per-preflight budget size-driven rejections recover exactly like empty-200 summaries already do
4 result.failure after a successful fold following an earlier rejection stale failure could be reported even when the payload fits or exhaustion has its own reason cleared; only the current end state is reported accurate diagnosis

Dead-end cooldown semantics are unchanged: it still arms only on zero-progress failures — but a 400 now reaches that state only after the FULL shrink cascade is exhausted, which is what makes it genuine evidence (the issue's fix #5, resolved behaviorally without key surgery).

Deferred (owner decision)

The issue's fix #4 (learned per-endpoint summary input ceiling in session metadata) adds a persistence/metadata surface — owner-design territory per the config-surface discipline, so it is tracked separately instead of being built here.

Tests

  • New tests/issue1993-preflight-400.test.ts: systemic hard 400 (snippet in client message AND warn log, bounded cascade ≥2 calls ≤ budget, nothing forwarded, cooldown arms with scoped key and replays with ZERO calls), size-driven 400 recovering via halving (end-to-end 200, failed-before-succeeded ordering, no marker left), 401 not shrinking (exactly one summary call), e2e dump-file assertion (envelope carries request+response bytes).
  • tests/error-dump.test.ts: off-by-default gate, envelope round-trip, raw non-JSON sides, cap + truncation marker, empty-exchange skip for dumpSummaryRejection.
  • Targeted runs green locally: every preflight-* suite plus error-dump (117 tests); typecheck + tsup build clean. Full regression is CI's job.

Closes #1993

@github-actions

github-actions Bot commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-03_preflight-400-recovery (8a24c9e)

Option A — Install this exact build (recommended)

npm install -g billion-context@0.0.2009-pr.1994.c39b99d

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr1994.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 收到,开始 review #1994(对应原 issue #1993):拉取分支核对代码、审既有行为与测试有效性、确认 CI 状态,稍后给结构化结论。

@ranxianglei

Copy link
Copy Markdown
Owner Author

On the single red check test (windows-latest, 24) — pre-existing flake on master, unrelated to this diff.

The failure is the freshly-merged perf pin tests/fold-reconcile-perf.test.ts ("cold rounds scale near-linearly to 8K messages / ~40MB", threshold t8k/t1k < 12). This PR touches no fold-reconcile code path.

Evidence it is pre-existing:

  • Master's own CI failed the identical job today: run 37119007536 (push c2e74acf, 2026-10-03 11:14Z, before this PR existed) — same test, same assertion: ratio 12.74 is super-linear (t1k=10.1ms t8k=129.0ms).
  • This PR's run: ratio 14.90 (t1k=10.7ms t8k=159.3ms). Every other job passed (ubuntu 22/24, windows 22); only windows-latest/node-24 fails, in both runs.

Likely mechanism: the warmup reconciles a 1K-message history, so the measured 8K rounds pay first-allocation cost for a size class never seen before. Min-of-3 absorbs GC/JIT spikes but not one-off page-fault/allocation cost of a fresh ~40MB working set, which appears much more variable on Windows runners (both failed runs show an elevated ~10ms t1k baseline for 1K messages).

Failed job re-run requested on this PR. Suggested follow-ups (different mechanism than #1993, out of scope here — please file/decide separately): warm up at the measured size class before taking min-of-3, or make the ratio pin OS-aware.

ework-agent added 2 commits October 3, 2026 21:17
…d 400/413 (#1993)

A preflight summarization call answered HTTP 400 left the session wedged and
undiagnosable: the response body was never logged or dumped, there was no
chunk-shrink fallback (unlike the unusable-HTTP-200 path), and the turn
fail-fasted 502 retryable=false plus a dead-end cooldown.

- surface the rejection body: bounded 200-char snippet in the warn log AND in
  the client-visible fail-fast detail (parity with the main request paths)
- BILI_DUMP_4XX now persists the full summary exchange (request + response)
  as summary-err-*.json under the same gate/cap; separate prefix so it cannot
  clobber a main-path err-* dump in the same millisecond
- hard (non-transient) 400/413 get the same halving recovery as an unusable
  200 summary, bounded by MAX_SUMMARY_CALLS_PER_PREFLIGHT; transient 4xx keep
  replay-retry semantics, auth/routing 4xx fail fast without burning budget
- clear a stale upstream failure after a successful fold so later exhaustion
  reports its own reason and fitting results report none

Closes #1993
@ranxianglei
ranxianglei force-pushed the 2026-10-03_preflight-400-recovery branch from ff0d0d1 to 8a24c9e Compare October 3, 2026 13:19
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 Review 完成(拉分支逐行核对代码 + 测试有效性 + CI + 对当前 master 的 rebase 验证)。

结构化七问

  1. 是什么问题:preflight 的 summary 调用被上游硬拒(400/413)时,拒绝 body 不记日志、不落盘,且没有 unusable-200 已有的减半重试路径——一次拒绝即 fail-fast 502 + 300s dead-end 冷却,会话卡死且无从诊断(preflight: summary call answered HTTP 400 — the response body is never logged or dumped, there is no chunk-shrink fallback, and the turn fail-fasts 502 retryable=false #1993 事故:~523K-token 输入连续 3 次 400,零证据留下)。
  2. 什么原因导致的:三层机制缺口——① requestSummaryBody 只记 status/attempts 不记 body;② walk 循环的 catch 分支里 UpstreamHttpError 4xx 一律走 fail-fast,而减半级联只存在于 HTTP-200-unusable 路径(两条恢复路径不对称);③ 无对应 dump 通道(err-*.json 只覆盖主转发请求)。
  3. 怎么修复的:① warn 日志 + client-visible detail 带 200 字符有界 body snippet(safePrefix,surrogate-safe,与 loop/core、server 主路径口径一致);② 新增 dumpSummaryRejection,挂在既有 BILI_DUMP_4XX 开关下,envelope {status, request, response} 恒为合法 JSON(request 侧是精确发送串,response 侧可解析则 JSON 化),单侧 cap 默认 2MiB;③ 非 transient 硬 400/413 且 span 可分 → 进入与 unusable-200 同一 halving worklist;transient 4xx 保留 replay-retry、其他 4xx(鉴权/路由)fail-fast 不烧预算、不可分 span 照旧放弃;fold 成功后清掉已记录的 failure,避免污染后续 exhausted 报告。
  4. 是否有回归问题:逐项审过既有行为,未发现回归——transient 4xx 语义未变(既有测试「temporary summary HTTP 400 failure does not cache a deterministic dead end」钉住);dead-end/冷却触发条件未变(仍需 compressedRanges===0 + exhausted/非重试 4xx,部分进度抑制冷却的既有条件原样);relaxed-regime 重置条件未变(failure || budgetHit 均阻断,preflight fail-fast × soft-protection livelock: over-window payload with nothing foldable outside the protected zone 502s forever #330 保护不变);wire fidelity 未动(不改任何转发 payload/tool args,client-visible detail 纯增量,body 为空时与旧文案逐字节一致);两种压缩模式(plugin/proxy)都是服务端公共路径,均覆盖。级联有界性已验证:continue 路径回到 while-top 的 budgetHit 检查(src/preflight.ts:1029/1063),预算耗尽不会无限分裂。
  5. 是否破坏缓存:不涉及——不动 system anchor/head 字节,前缀缓存相关行为零变化。
  6. 是否改了设计方向:不涉及。issue 建议 feat: OpenAI Responses API support (/v1/responses) #4(per-endpoint 学习式上限入 session metadata)作者正确 defer(config surface / owner 领域),PR 范围干净。
  7. 是否引入新配置:无——dump 复用既有 BILI_DUMP_4XX / BILI_DUMP_4XX_MAX_BYTES(该开关此前也未在 CONFIGURATION.md 记载,前后一致,不算本 PR 引入的不一致)。

行为变更披露(旧→新):硬 400/413 时每次 preflight 调用的上游 summary 调用数:旧 = 恰好 1 次 → 新 = 最多到该次调用的 summary 预算。注意与 #1933 的叠加:master 上 #1933 已把固定 16 改成随 overshoot 动态的 summaryBudget(16→至多 32),所以合并后实际最坏情况是 ≤32(超窗大 payload)/ ≤16(常规),不是 PR 描述写的 ≤16——这与 #1933 的设计意图一致(越大越给恢复空间),但披露值要按此理解。

我做的验证动作:

判定:可合 —— 等 8a24c9e 的 CI 转绿后人工合并即可。(自动合并门禁不过:运行时默认行为变更属产品决策,留在人工。)

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 CI 更新:rebase 后的新 head 8a24c9e 全矩阵 14/14 绿,包括之前 flake 的 test (windows-latest, 24)(fold-reconcile-perf 这次过了)。无新增失败。

结论维持上一条:可合 —— 现在可以直接人工合并 PR #1994(CI 已绿)。

@ranxianglei
ranxianglei merged commit 129200d into master Oct 3, 2026
14 checks passed
ranxianglei pushed a commit that referenced this pull request Oct 3, 2026
ranxianglei pushed a commit that referenced this pull request Oct 3, 2026
…e-narrow in #1987/#1994/ws-bridge tests

- src/server.ts: re-export type ProxyOptions (issue1987/issue1993 tests import it from server.ts)
- tests/issue1987-preflight-baseline-anchor.test.ts: address() as AddressInfo (×2)
- tests/codex-responses-ws.test.ts: cast tools/input/error.error (×4 errors, type-level only)
- tests/ws-bridge.test.ts: async void fallback handler + Promise<void> resolve

Touched test files pass: 5+1+5+3. No runtime changes anywhere.
ranxianglei added a commit that referenced this pull request Oct 4, 2026
fix(typecheck): unblock CI — export ProxyOptions + type-narrow #1987/#1994/ws-bridge tests
@github-actions github-actions Bot mentioned this pull request Oct 4, 2026
@ranxianglei
ranxianglei deleted the 2026-10-03_preflight-400-recovery branch October 15, 2026 05:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant