Skip to content

fix(preflight): fold deep enough for a payload far over the window (#868) - #869

Merged
ranxianglei merged 7 commits into
ranxianglei:masterfrom
joshwapohlmann:2026-09-16_868-preflight-round-budget
Sep 17, 2026
Merged

ranxianglei merged 7 commits into
ranxianglei:masterfrom
joshwapohlmann:2026-09-16_868-preflight-round-budget

Conversation

@joshwapohlmann

Copy link
Copy Markdown
Contributor

Problem

The preflight folds one range per round, with both per-invocation budgets at 8. A payload far beyond the window therefore fails even when every fold succeeds, and the failure reads as a total one:

preflight compressed 8 range(s), ~329022 tokens saved (1385572 -> 1056550)
preflight fail-fast 502 preflight_compress_failed: the compress budget was exhausted after 8 rounds

That conversation was folded to within 8k tokens of a 1,048,576 window and then declared exhausted. The message also quotes the original context, so a reader sees 1.39M tokens and no folding, and the session takes the preflight dead-end cooldown on top.

Fix

MAX_PREFLIGHT_ROUNDS and MAX_SUMMARY_CALLS_PER_PREFLIGHT in src/preflight.ts are raised to 16, and the latter is exported so the budget-cap tests derive it instead of pinning a literal. Nothing else changes: a round still folds one range, and a payload already over 16 ranges wide still stops and reports, but at a depth that matches a window-sized payload rather than a marginally oversized one.

Tests

tests/preflight-round-budget.test.ts drives the fold loop over an openai-wire fixture of roughly 83k tokens against a 30k window. It refuses the request at 8 rounds (preflight_compress_failed) and forwards it at 16, using more than 8 summary calls on the way, which pins the behaviour rather than the constant. The #574 cap test in tests/preflight-multi-range.test.ts now takes its expected number from the exported constant instead of re-pinning 8, so the two cannot drift apart again.

Verification

Full proxy suite on the branch: 1669 tests, 1661 passing, the same six environmental failures upstream shows. npx tsc --noEmit -p tsconfig.build.json clean. The live case that produced the report now folds further and forwards instead of failing; the numbers above are from that session, and the test fixture reproduces the same shape at a smaller scale so the behaviour is pinned without needing a 1.39M-token fixture in the suite.

Joshwa Pohlmann added 3 commits September 16, 2026 18:32
A preflight folds one range per round and can spend a bounded number of
summarization calls, so a payload that is well over the window and split into
many ranges needs several rounds. Eight of each was not enough for the cases
that matter: a live Gemini session of 1.39M tokens against a 1.05M window had
every fold it attempted succeed and still came out ~8k short, then dropped the
turn with "the compress budget was exhausted after 8 rounds" — the session was
recoverable and the budget refused to recover it.

Both budgets move to 16, which the new test pins: a fixture whose fold needs
more rounds than the old budget allows is refused at 8 and forwarded at 16
(needs a fold deeper than one round, and stays bounded). The preflight is the
last resort before an outright failure, so a bounded doubling of its work is
cheaper than the failure it prevents.
@joshwapohlmann

Copy link
Copy Markdown
Contributor Author

I would like both, in this order, and you are welcome to add them here.

  1. The honest failure detail. Make the exhaustion path and failFast report the reduced post-fold token count and the remaining compressible-range count instead of the original tokenCount (src/server.ts:2862 reading the value captured at src/server.ts:2831). That is the half that changes what an operator reads at 502 time, and it is the half of the issue's "Expected" this PR does not yet meet. A commit on this branch is the right place while it is unmerged.

  2. The budget: document, do not redesign yet. If the coverage bound can be stated beside the constants (how far over the window 16 folds reaches, and that beyond it the loop still exits with ranges left), that is enough. A budget scaled to the overshoot is a design change I would rather see reviewed on its own, and your point that 16 is a number tuned to this incident is right.

Neither becomes an issue: the filing rule for this project is pull requests only, because an issue in either repository sets the review agent implementing in parallel — which is what happened on the kernel's #319. If you would rather do 2 as a design change, say so and I will file it as its own PR instead.

Thanks for confirming the root cause and for deriving the test from the exported constant instead of pinning 8.

ework-agent added 2 commits September 17, 2026 08:17
… fail-fast 502

The exhaustion path quoted the original tokenCount captured before folding,
so a 502 after 16 successful folds read as if nothing happened (the ranxianglei#868
incident: 1.39M -> 1.06M, still over the 1M window). The fail-fast now
reports the post-fold size ("~N tokens (down from ~M before preflight)") and
the compressible ranges still visible, via PreflightResult.rangesRemaining
tracked per round from the kernel's viable ranges; original phrasing is kept
verbatim when no fold ran. MAX_PREFLIGHT_ROUNDS is exported next to
MAX_SUMMARY_CALLS_PER_PREFLIGHT so tests derive budgets from constants.
New e2e case scales the round-budget fixture past the bound and pins the
honest 502 (round-budget wording, post-fold size, remaining-range count,
exactly one summary call per round, nothing forwarded).
Document next to MAX_PREFLIGHT_ROUNDS / MAX_SUMMARY_CALLS_PER_PREFLIGHT how
far over the window 16 folds can reach (one range per round x at most
CHUNK_FRACTION of the window per fold ~= 9.6x-window overshoot, i.e. a payload
up to ~10.6x the window, best case; lower in practice) and that beyond the
bound the loop still exits cleanly with the post-fold size and remaining
compressible-range count reported. Note that 16 is tuned to the ranxianglei#868 incident
class — an overshoot-scaled budget is deliberately left as a separate design
question.
@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 Both items are done and pushed to your branch joshwapohlmann/billion-context@2026-09-16_868-preflight-round-budget (ea17a12..0a16f29) as two commits. (Heads-up: my session hit a host-level disk-full mid-task earlier today — I posted a blocked status; everything below is now verified.)

1. Honest failure detail — e05894e fix(preflight): report the post-fold size and remaining ranges in the fail-fast 502

  • PreflightResult.rangesRemaining (src/preflight.ts): compressible ranges still visible in the kernel's final view after the walk stopped, captured each round from the viable ranges.
  • The fail-fast 502 (src/server.ts, the closure you pointed at) now reads, when folding happened:
    context ~<post-fold> tokens (down from ~<original tokenCount> before preflight) exceeds the model window <limit> (model=<m>), with <N> compressible range(s) still visible and preflight compression should not have been needed… — i.e. for the filed incident it would say ~1,056,550 tokens (down from ~1,385,572 before preflight) … with <N> compressible range(s) still visible … exhausted after 16 rounds. Post-fold size is the loop's own currentTokens (same number the existing "tokens saved (A → B)" log prints). When no fold ran, the original phrasing is kept verbatim, so no existing message shape regresses; the exhaustion detail strings are untouched, which keeps the /summarization budget/i and /no range could be compressed/i pins green.
  • MAX_PREFLIGHT_ROUNDS is now exported next to MAX_SUMMARY_CALLS_PER_PREFLIGHT, so tests derive budgets from constants instead of pinning literals.
  • New e2e case in tests/preflight-round-budget.test.ts: the fixture scaled past the bound (400 turns, ~340k tokens vs a 30k window) → asserts 502 + "exhausted after 16 rounds" + "down from … before preflight" + "… compressible range(s) still visible" + exactly one summary call per round + nothing forwarded. One matching assertion added to the fix: preflight tries all viable ranges before declaring exhaustion (#569) #574 budget-cap case in tests/preflight-multi-range.test.ts (zero-fold path).

2. Coverage bound documented, not redesigned — 0a16f29 docs(preflight): state the fold-depth coverage bound beside the budgets
Comment beside the constants in src/preflight.ts: one range per round × ≤ CHUNK_FRACTION×window removed per fold ⇒ 16 rounds cover at most ~9.6×window of overshoot (payload up to ~10.6×window, best case); real coverage lower (sub-budget ranges save less, #726 halving can spend several calls on one range); beyond the bound the loop exits cleanly with the post-fold size + remaining-range count reported. It also states plainly that 16 is tuned to the #868 incident class and that an overshoot-scaled budget is left as a separate design question — I did not touch the mechanism. No new issues or PRs filed, per your rule.

Verification: npm run typecheck clean; full suite 1651 tests / 0 failures / 2 skipped (the gate-only e2e) — the 6 environmental failures from your run did not reproduce here; npm run build clean. PR: #869

中文摘要:按你的顺序完成了两项——502 报错现在报告折叠后 token 数(含折叠前原值对比)与剩余可压缩区间数,覆盖边界文档写在常量旁而未改动预算机制;两个提交已推送到你的分支,typecheck、全量测试(1651 通过 0 失败)和构建均已验证。

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 Correction to my previous comment: I garbled the quoted message format. The exact shape of the new 502 body (for the filed incident's numbers) is:

context ~1,056,550 tokens (down from ~1,385,572 before preflight) exceeds the model window 1,048,576 (model=<m>), with <N> compressible range(s) still visible and preflight compression could not bring it under: the compress budget was exhausted after 16 rounds. The over-window payload was NOT forwarded.

Everything else in the report stands unchanged.

@ranxianglei

Copy link
Copy Markdown
Owner

解决冲突

@ranxianglei

Copy link
Copy Markdown
Owner

继续

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 Conflict resolved and pushed. The branch joshwapohlmann/billion-context@2026-09-16_868-preflight-round-budget now has master (up to 55da3f0, release v0.1.117) merged in as one commit eb6c475, and GitHub reports the PR mergeable / clean.

What conflicted. A single hunk in src/preflight.ts (the end of preflightCompress). This branch inserts its new result.rangesRemaining = rangesRemaining; line directly after if (result.compressedRanges > 0) deps.session.stats.lastInputTokens = currentTokens; — and master's #857 (cd0cd88) rewrote exactly that line into the image-floor-aware baseline block. Everything else auto-merged, including src/server.ts (the fail-fast message change lands cleanly on top of master's changes).

Resolution. Kept master's #857 block verbatim and appended this branch's rangesRemaining assignment after it. Rationale: the old one-liner was base context that #857 deliberately superseded (never persist the image floor into the usage baseline; raise-only with lastInputTokensSource = "estimate"), not a change made by this PR — so the union of intent is "master's baseline semantics + this branch's new field". No behaviour from either side was dropped.

Verification on the merged head (eb6c475):

  • npm run typecheck — clean
  • npm test — 1672 tests, 1670 passing, 0 failing, 2 skipped (the gate-only e2e)
  • npm run build — success
  • GitHub API: mergeable: true, mergeable_state: clean

No other commits were touched; the five branch commits keep their hashes, so everything in my earlier report still applies. Ready for you to merge when you are.

中文摘要:解决了分支与 master(至 v0.1.117)在 src/preflight.ts 末尾的唯一冲突——保留 master #857 的基线逻辑、追加本分支新增的 rangesRemaining 赋值,合并提交 eb6c475 已推送;typecheck/全量测试(1670 通过)/构建全部通过,PR 现显示可干净合并。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants