Repository navigation
fix: count system+tools wire overhead in the preflight trigger (#470) - #576
Conversation
The preflight trigger and fit gates estimated the payload with estimateCoreMessages (messages only), while the billed input also carries the system prompt and tool definitions — including the proxy-injected compress system prompt (~1.9k tokens) and ACP tools (~0.5k), plus whatever the client sends (~10-20k on agent clients). The trigger fired late: text alone 'fit' the window while the real input already overflowed it, and the post-fold fit gate could pass on payloads whose billed input still exceeded the window. - server.ts: estimateWireOverhead(protocol, body) — counts system/instructions text (string, array blocks, openai hoisted system/developer messages) + tools JSON from the rebuilt wire body - preflightCompressIfNeeded: trigger and both fit gates add the term - preflight.ts: PreflightDeps.wireOverhead — the fold loop's currentTokens/payloadEstimate floors include it so the loop keeps folding until the true billed input fits (was: stopped at 'text fits' then fail-fasted against the caller's overhead-aware gate) - README: the plugin-mode preflight cell said 'No' — preflight runs unconditionally in BOTH modes as the last-resort backstop (#470) Tests: e2e through the proxy (text fits alone, system+tools push it over -> preflight folds and forwards); #330 and #453 e2e re-tuned for the honest accounting (their payloads previously forwarded while the real billed input exceeded the window); 6 unit tests for the helper.
📦 Built Package ArtifactBranch: Option A — Install from npm PR tag (recommended)npm install -g billion-context@pr-576Each push to this PR publishes a new version under the Option B — Download artifact
tar xzf billion-context-pr576.tgz
npm install -g packageThis comment is automatically updated on each push. |
[bot] 🏷 Done — opened PR #577: #577 (branch What changed
Verification
One clarification for your call: the issue's "#330 e2e re-tuned (24k chars instead of 32k)" has no matching test by that name in the tree (grep found none — only an unrelated Ready for review — not merging per repo rules. |
[bot] 🏷 Closed as duplicate of #577 (per @ranxianglei's cross-ref on #577: one of the two PRs implementing the #470 fix should be closed so the issue resolves through a single PR). Why #577 was kept over this one:
Tagged |
Closes #470 (residual gap from the analysis in the issue thread).
Root cause
The preflight trigger and fit gates estimated the payload with
estimateCoreMessages(messages only), but the billed input also carries the system prompt and tool definitions — the client's own (~10-20k on agent clients) plus the proxy-injected compress system prompt (~1.9k tokens) and ACP tools (~0.5k). Consequences:#467 fixed this class of under-count only for the output clamp (
estimateInputTokens); this PR applies the same term to preflight — the plugin-mode last resort.Change
src/server.ts: newestimateWireOverhead(protocol, body)— counts system/instructions text (anthropic string/array forms, openai hoisted system/developer messages, responses instructions) + tools JSON from the rebuilt wire body, viadefaultCountTokens.preflightCompressIfNeeded: trigger + both post-fold fit gates add the term.src/preflight.ts:PreflightDeps.wireOverhead— the fold loop'scurrentTokens/payloadEstimatefloors include it, so the loop keeps folding until the true billed input fits. Without this the loop stopped at "text fits" and then fail-fasted against the caller's overhead-aware gate (502 on foldable payloads).README.md: the two-mode table's plugin-mode preflight cell said "No" — preflight runs unconditionally in both modes as the last-resort backstop; corrected.Test retuning (behavior change is intended)
tests/preflight-fail-fast.test.ts: new e2e — text fits the 10k window alone, system+tools push the billed input over → preflight folds and forwards; forwarded body no longer carries the raw big message. The preflight fail-fast × soft-protection livelock: over-window payload with nothing foldable outside the protected zone 502s forever #330 e2e re-tuned (24k chars instead of 32k): its old payload was forwarded while the real billed input exceeded the window.tests/issue453-backstop.test.ts: the clamp e2e re-sized (window 400k / max_tokens 150k) to sit in the band where the clamp acts alone — the old shape now correctly triggers preflight (fold instead of output starvation), which is the better outcome.tests/wire-overhead.test.ts: 6 unit tests for the helper (protocol matrix, unparseable body → 0).Pre-flight
npm run typecheck✅npm test1070/1072 — the only failures are the 2 known env-dependent ones inplugin-agent.test.ts(fail locally on this machine, pass in CI) ✅npm run build✅Out of scope (per the issue analysis)