Skip to content

fix: count system+tools wire overhead in the preflight trigger (#470) - #576

Merged
ranxianglei merged 1 commit into
masterfrom
2026-09-06_preflight-wire-overhead
Sep 6, 2026
Merged

ranxianglei merged 1 commit into
masterfrom
2026-09-06_preflight-wire-overhead

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

Closes #470 (residual gap from the analysis in the issue thread).

Root cause

The preflight trigger and fit gates estimated the payload with estimateCoreMessages (messages only), but the billed input also carries the system prompt and tool definitions — the client's own (~10-20k on agent clients) plus the proxy-injected compress system prompt (~1.9k tokens) and ACP tools (~0.5k). Consequences:

  • the trigger fired late: text alone "fit" the window while the real input already overflowed it (guaranteed upstream 400 on strict relays);
  • the post-fold fit gate could pass on payloads whose billed input still exceeded the window.

#467 fixed this class of under-count only for the output clamp (estimateInputTokens); this PR applies the same term to preflight — the plugin-mode last resort.

Change

  • src/server.ts: new estimateWireOverhead(protocol, body) — counts system/instructions text (anthropic string/array forms, openai hoisted system/developer messages, responses instructions) + tools JSON from the rebuilt wire body, via defaultCountTokens.
  • preflightCompressIfNeeded: trigger + both post-fold fit gates add the term.
  • src/preflight.ts: PreflightDeps.wireOverhead — the fold loop's currentTokens/payloadEstimate floors include it, so the loop keeps folding until the true billed input fits. Without this the loop stopped at "text fits" and then fail-fasted against the caller's overhead-aware gate (502 on foldable payloads).
  • README.md: the two-mode table's plugin-mode preflight cell said "No" — preflight runs unconditionally in both modes as the last-resort backstop; corrected.

Test retuning (behavior change is intended)

  • tests/preflight-fail-fast.test.ts: new e2e — text fits the 10k window alone, system+tools push the billed input over → preflight folds and forwards; forwarded body no longer carries the raw big message. The preflight fail-fast × soft-protection livelock: over-window payload with nothing foldable outside the protected zone 502s forever #330 e2e re-tuned (24k chars instead of 32k): its old payload was forwarded while the real billed input exceeded the window.
  • tests/issue453-backstop.test.ts: the clamp e2e re-sized (window 400k / max_tokens 150k) to sit in the band where the clamp acts alone — the old shape now correctly triggers preflight (fold instead of output starvation), which is the better outcome.
  • tests/wire-overhead.test.ts: 6 unit tests for the helper (protocol matrix, unparseable body → 0).

Pre-flight

  • npm run typecheck ✅
  • npm test 1070/1072 — the only failures are the 2 known env-dependent ones in plugin-agent.test.ts (fail locally on this machine, pass in CI) ✅
  • npm run build ✅

Out of scope (per the issue analysis)

  • The optional stateless request-scoped drop (deep-tail plugin-mode backstop) — not needed while the trigger is honest; can be revisited if real-world reports show inputs that fit preflight's accounting but still 400.

The preflight trigger and fit gates estimated the payload with
estimateCoreMessages (messages only), while the billed input also
carries the system prompt and tool definitions — including the
proxy-injected compress system prompt (~1.9k tokens) and ACP tools
(~0.5k), plus whatever the client sends (~10-20k on agent clients).
The trigger fired late: text alone 'fit' the window while the real
input already overflowed it, and the post-fold fit gate could pass on
payloads whose billed input still exceeded the window.

- server.ts: estimateWireOverhead(protocol, body) — counts
  system/instructions text (string, array blocks, openai hoisted
  system/developer messages) + tools JSON from the rebuilt wire body
- preflightCompressIfNeeded: trigger and both fit gates add the term
- preflight.ts: PreflightDeps.wireOverhead — the fold loop's
  currentTokens/payloadEstimate floors include it so the loop keeps
  folding until the true billed input fits (was: stopped at 'text
  fits' then fail-fasted against the caller's overhead-aware gate)
- README: the plugin-mode preflight cell said 'No' — preflight runs
  unconditionally in BOTH modes as the last-resort backstop (#470)

Tests: e2e through the proxy (text fits alone, system+tools push it
over -> preflight folds and forwards); #330 and #453 e2e re-tuned for
the honest accounting (their payloads previously forwarded while the
real billed input exceeded the window); 6 unit tests for the helper.
@github-actions

github-actions Bot commented Sep 6, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-09-06_preflight-wire-overhead (8e0a6ad)

Option A — Install from npm PR tag (recommended)

npm install -g billion-context@pr-576

Each push to this PR publishes a new version under the pr-576 npm tag.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr576.tgz
npm install -g package

This comment is automatically updated on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 ework agent · qwen3.8-27b

[bot] 🏷 Done — opened PR #577: #577 (branch wt-576-ework-daemon → master).

What changed

  • src/server.ts: new estimateWireOverhead(protocol, body) counts system/instructions text (anthropic string/array forms, openai hoisted system/developer messages, responses instructions) + tools JSON from the rebuilt wire body via defaultCountTokens (unparseable → 0; text-only so it never double-counts image tokens). Wired into the preflight trigger and both post-fold fit gates.
  • src/preflight.ts: PreflightDeps.wireOverhead floors the fold loop's currentTokens/payloadEstimate, so it keeps folding until the true billed input fits (previously it stopped at "text fits" and then fail-fasted against the caller's now overhead-aware gate → 502 on foldable payloads).
  • README.md: corrected the two-mode table's plugin-mode preflight cell ("No" → runs unconditionally in both modes as the last-resort backstop).

Verification

  • New tests/wire-overhead.test.ts (6 unit tests) + a new e2e proving envelope-driven overflow folds & forwards — confirmed it fails on master (skips preflight entirely) and passes here.
  • Retuned the affected fixtures to sit under honest accounting: codex-compact harness window 10k→12k (setup turn's billed ≈10.1k stays under while bigConversation() stays over), finality turn-1 filler trimmed, issue453 clamp re-sized to window 400k / max_tokens 150k so the clamp acts alone.
  • npm run typecheck ✅ · npm run build ✅ · npm test 1070/1071 — the single failure is a pre-existing, environment-dependent CLI-launcher test (resolveClientCommand: codex/claude resolve to themselves), present on baseline master and unrelated to this change.
  • Gated E2E regression (tests/e2e/e2e-codex.test.ts, ACP_TEST_E2E=1) couldn't run in this sandbox (no live upstream at the default local address; codex-cli present, dist built, zero-token preflight check passed). CI's manual-dispatch run covers it.

One clarification for your call: the issue's "#330 e2e re-tuned (24k chars instead of 32k)" has no matching test by that name in the tree (grep found none — only an unrelated windowEnv(32_000) in the gated e2e). That behavior is covered by the new #470 e2e, so I didn't retune a separate test. If you had a specific existing test in mind, point me at it and I'll adjust.

Ready for review — not merging per repo rules.

@ranxianglei ranxianglei closed this Sep 6, 2026
@ranxianglei ranxianglei added the duplicate This issue or pull request already exists label Sep 6, 2026
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 ework agent · qwen3.8-27b

[bot] 🏷 Closed as duplicate of #577 (per @ranxianglei's cross-ref on #577: one of the two PRs implementing the #470 fix should be closed so the issue resolves through a single PR).

Why #577 was kept over this one:

  1. Clean base. fix: count system+tools wire overhead in the preflight trigger (#470) #577's head (3b746a5) is a single commit directly on current master (a184ff9) — merging it lands exactly the track: plugin-mode proxy-side emergency truncation (last-resort backstop, deferred from #467) #470 fix and nothing else. This PR's head (8e0a6ad) carries the orphaned fix: preflight relaxes the soft-protected recent zone under overflow (#330) #337 merge commit (6820e7d) in its history; since current master does not contain fix: preflight relaxes the soft-protected recent zone under overflow (#330) #337's code, merging this PR would silently re-land the preflight fail-fast × soft-protection livelock: over-window payload with nothing foldable outside the protected zone 502s forever #330 soft-zone relax + PR #489 后续: 默认图片计费 b64/4 在像素 tile 计费上游(官方 Anthropic/OpenAI)会硬 502 误杀多模态会话 #496 image-cost fix alongside the track: plugin-mode proxy-side emergency truncation (last-resort backstop, deferred from #467) #470 fix, undisclosed (the body only says "Closes track: plugin-mode proxy-side emergency truncation (last-resort backstop, deferred from #467) #470"). Master was deliberately rolled back after fix: preflight relaxes the soft-protected recent zone under overflow (#330) #337's merge today — if fix: preflight relaxes the soft-protected recent zone under overflow (#330) #337 should land, do it as its own disclosed PR (the code is intact at 6820e7d / branch 2026-08-29_preflight-soft-zone-relax).
  2. More complete iteration. This PR's description claims retunes of tests/codex-compact-e2e.test.ts (window 10k→12k) and tests/compaction-trigger-finality.test.ts (trimmed turn-1 filler) — but neither file is in this diff. fix: count system+tools wire overhead in the preflight trigger (#470) #577's diff does contain those retunes.
  3. Both heads verified green locally anyway (typecheck clean; full suite fails only the known env-dependent resolveClientCommand launcher test, reproduced identically on master baseline), so no work is lost by closing this one.

Tagged duplicate. Reopen if you decide the #337 bundling was actually intended.

@ranxianglei ranxianglei reopened this Sep 6, 2026
@ranxianglei
ranxianglei merged commit a905527 into master Sep 6, 2026
12 checks passed
@ranxianglei
ranxianglei deleted the 2026-09-06_preflight-wire-overhead branch October 15, 2026 05:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

duplicate This issue or pull request already exists

Projects

None yet

Development

Successfully merging this pull request may close these issues.

track: plugin-mode proxy-side emergency truncation (last-resort backstop, deferred from #467)

1 participant