Repository navigation
fix: count system+tools wire overhead in the preflight trigger (#470) - #577
ranxianglei wants to merge 1 commit into
Conversation
The preflight trigger and post-fold fit gates estimated the payload with estimateCoreMessages (messages only), but the billed input also carries the system prompt and tool definitions (~10-20k on agent clients plus the proxy-injected compress prompt ~1.9k and ACP tools ~0.5k). Consequences: - the trigger fired late: text alone "fit" the window while the real input already overflowed it (guaranteed upstream 400 on strict relays); - the post-fold fit gates could pass payloads whose billed input still exceeded the window. #467 fixed this class of under-count only for the output clamp (estimateInputTokens); this applies the same term to preflight, the plugin-mode last resort. Changes: - src/server.ts: new estimateWireOverhead(protocol, body) counts system/instructions text (anthropic string/array, openai hoisted system/developer messages, responses instructions) + tools JSON from the rebuilt wire body via defaultCountTokens. Unparseable -> 0. Text-only, so it never double-counts image tokens. - preflightCompressIfNeeded: trigger + both post-fold fit gates add the term; the fold loop receives it as PreflightDeps.wireOverhead so currentTokens / payloadEstimate floors include it and the loop keeps folding until the true billed input fits. - README.md: two-mode table plugin-mode preflight cell said "No"; preflight runs unconditionally in both modes as the last-resort backstop. Tests: new tests/wire-overhead.test.ts (6 unit tests); new e2e proving envelope-driven overflow folds + forwards; affected fixtures re-sized to sit under honest accounting. typecheck + build clean; full suite green except one pre-existing env-dependent failure unrelated to this change.
[bot] 🏷 Review complete — code verified, tests reproduced. One item needs your confirmation before merge; one description nit. Verdict: the fix is correct and complete. I checked out head What I validated
|
|
Cross-ref: #576 implements the same #470 fix (same preflight wire-overhead root cause, opened ~1h earlier from the issue-thread analysis). Both are currently open against master — one of the two should be closed so #470 resolves through a single PR. #576 is mergeable against current master (53ce3ba) with pre-flight typecheck/test/build green (1067/1069 pass, 2 known plugin-agent env fails). |
[bot] 🏷 Dedup done: kept #577, closed #576 as duplicate (tagged What I foundThe two PRs are two independent implementations of the same fix, not one branch pushed twice:
Note: my earlier review in this thread was of
Why #577 wins
Two notes
Everything else green on my side. Merge stays human-only per repo rules — ready when you are. |
Closes #470 (residual gap from the analysis in the issue thread).
Root cause
The preflight trigger and fit gates estimated the payload with
estimateCoreMessages(messages only), but the billed input also carries the system prompt and tool definitions — the client's own (~10-20k on agent clients) plus the proxy-injected compress system prompt (~1.9k tokens) and ACP tools (~0.5k). Consequences:#467 fixed this class of under-count only for the output clamp (
estimateInputTokens); this PR applies the same term to preflight — the plugin-mode last resort.Change
src/server.ts: newestimateWireOverhead(protocol, body)— counts system/instructions text (anthropic string/array forms, openai hoisted system/developer messages, responses instructions) + tools JSON from the rebuilt wire body, viadefaultCountTokens. Unparseable → 0; text-only so it never double-counts image tokens.preflightCompressIfNeeded: trigger + both post-fold fit gates add the term.src/preflight.ts:PreflightDeps.wireOverhead— the fold loop'scurrentTokens/payloadEstimatefloors include it, so the loop keeps folding until the true billed input fits. Without this the loop stopped at "text fits" and then fail-fasted against the caller's overhead-aware gate (502 on foldable payloads).README.md: the two-mode table's plugin-mode preflight cell said "No" — preflight runs unconditionally in both modes as the last-resort backstop; corrected.Test retuning (behavior change is intended)
tests/wire-overhead.test.ts(new): 6 unit tests for the helper (protocol matrix, unparseable body → 0, Buffer parity).tests/preflight-fail-fast.test.ts: new e2e —bigConversation()is 13.1k tokens, under a 15k window on text alone (master skips preflight) but the injected compress prompt + ACP tools push the billed input to ~15.5k → preflight folds and forwards; the forwarded body no longer carries the early message and does carry the folded summary. Verified this e2e fails on master (skips preflight entirely) and passes here.tests/codex-compact-e2e.test.ts: widened the harness window 10k→12k so the setup turn's honest billed input (text + envelope ≈ 10.1k) stays under the window whilebigConversation()remains over it — preserves master's exact message sizes and every forge/gate scenario.tests/compaction-trigger-finality.test.ts: trimmed turn-1 filler so its honest billed input sits just under the 10k window.tests/issue453-backstop.test.ts: the clamp e2e re-sized (window 400k / max_tokens 150k) to sit in the band where the clamp acts alone — the old shape now correctly triggers preflight (fold instead of output starvation), which is the better outcome.Note on the issue's "#330 e2e re-tuned (24k chars instead of 32k)": there is no distinct existing test by that name in the tree (grep found none); the behavior it describes is covered by the new
#470e2e above, so no separate retune was required.Pre-flight
npm run typecheck✅npm test1070/1071 — the single failure is a pre-existing, environment-dependent CLI-launcher resolution test (resolveClientCommand: codex/claude resolve to themselves), unrelated to this change and present on baseline master too.npm run build✅tests/e2e/e2e-codex.test.ts,ACP_TEST_E2E=1) could not run in this sandbox (no live upstream at the default local address; codex-cli present, dist built, zero-token preflight check passed). CI's manual-dispatch run covers it.Out of scope (per the issue analysis)