Skip to content

fix: count system+tools wire overhead in the preflight trigger (#470) - #577

Closed
ranxianglei wants to merge 1 commit into
masterfrom
wt-576-ework-daemon
Closed

ranxianglei wants to merge 1 commit into
masterfrom
wt-576-ework-daemon

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

Closes #470 (residual gap from the analysis in the issue thread).

Root cause

The preflight trigger and fit gates estimated the payload with estimateCoreMessages (messages only), but the billed input also carries the system prompt and tool definitions — the client's own (~10-20k on agent clients) plus the proxy-injected compress system prompt (~1.9k tokens) and ACP tools (~0.5k). Consequences:

  • the trigger fired late: text alone "fit" the window while the real input already overflowed it (guaranteed upstream 400 on strict relays);
  • the post-fold fit gate could pass on payloads whose billed input still exceeded the window.

#467 fixed this class of under-count only for the output clamp (estimateInputTokens); this PR applies the same term to preflight — the plugin-mode last resort.

Change

  • src/server.ts: new estimateWireOverhead(protocol, body) — counts system/instructions text (anthropic string/array forms, openai hoisted system/developer messages, responses instructions) + tools JSON from the rebuilt wire body, via defaultCountTokens. Unparseable → 0; text-only so it never double-counts image tokens.
  • preflightCompressIfNeeded: trigger + both post-fold fit gates add the term.
  • src/preflight.ts: PreflightDeps.wireOverhead — the fold loop's currentTokens/payloadEstimate floors include it, so the loop keeps folding until the true billed input fits. Without this the loop stopped at "text fits" and then fail-fasted against the caller's overhead-aware gate (502 on foldable payloads).
  • README.md: the two-mode table's plugin-mode preflight cell said "No" — preflight runs unconditionally in both modes as the last-resort backstop; corrected.

Test retuning (behavior change is intended)

  • tests/wire-overhead.test.ts (new): 6 unit tests for the helper (protocol matrix, unparseable body → 0, Buffer parity).
  • tests/preflight-fail-fast.test.ts: new e2e — bigConversation() is 13.1k tokens, under a 15k window on text alone (master skips preflight) but the injected compress prompt + ACP tools push the billed input to ~15.5k → preflight folds and forwards; the forwarded body no longer carries the early message and does carry the folded summary. Verified this e2e fails on master (skips preflight entirely) and passes here.
  • tests/codex-compact-e2e.test.ts: widened the harness window 10k→12k so the setup turn's honest billed input (text + envelope ≈ 10.1k) stays under the window while bigConversation() remains over it — preserves master's exact message sizes and every forge/gate scenario.
  • tests/compaction-trigger-finality.test.ts: trimmed turn-1 filler so its honest billed input sits just under the 10k window.
  • tests/issue453-backstop.test.ts: the clamp e2e re-sized (window 400k / max_tokens 150k) to sit in the band where the clamp acts alone — the old shape now correctly triggers preflight (fold instead of output starvation), which is the better outcome.

Note on the issue's "#330 e2e re-tuned (24k chars instead of 32k)": there is no distinct existing test by that name in the tree (grep found none); the behavior it describes is covered by the new #470 e2e above, so no separate retune was required.

Pre-flight

  • npm run typecheck ✅
  • npm test 1070/1071 — the single failure is a pre-existing, environment-dependent CLI-launcher resolution test (resolveClientCommand: codex/claude resolve to themselves), unrelated to this change and present on baseline master too.
  • npm run build ✅
  • Gated E2E regression (tests/e2e/e2e-codex.test.ts, ACP_TEST_E2E=1) could not run in this sandbox (no live upstream at the default local address; codex-cli present, dist built, zero-token preflight check passed). CI's manual-dispatch run covers it.

Out of scope (per the issue analysis)

  • The optional stateless request-scoped drop (deep-tail plugin-mode backstop) — not needed while the trigger is honest; can be revisited if real-world reports show inputs that fit preflight's accounting but still 400.

The preflight trigger and post-fold fit gates estimated the payload with
estimateCoreMessages (messages only), but the billed input also carries the
system prompt and tool definitions (~10-20k on agent clients plus the
proxy-injected compress prompt ~1.9k and ACP tools ~0.5k). Consequences:

- the trigger fired late: text alone "fit" the window while the real input
  already overflowed it (guaranteed upstream 400 on strict relays);
- the post-fold fit gates could pass payloads whose billed input still
  exceeded the window.

#467 fixed this class of under-count only for the output clamp
(estimateInputTokens); this applies the same term to preflight, the
plugin-mode last resort.

Changes:
- src/server.ts: new estimateWireOverhead(protocol, body) counts
  system/instructions text (anthropic string/array, openai hoisted
  system/developer messages, responses instructions) + tools JSON from the
  rebuilt wire body via defaultCountTokens. Unparseable -> 0. Text-only, so it
  never double-counts image tokens.
- preflightCompressIfNeeded: trigger + both post-fold fit gates add the term;
  the fold loop receives it as PreflightDeps.wireOverhead so currentTokens /
  payloadEstimate floors include it and the loop keeps folding until the true
  billed input fits.
- README.md: two-mode table plugin-mode preflight cell said "No"; preflight
  runs unconditionally in both modes as the last-resort backstop.

Tests: new tests/wire-overhead.test.ts (6 unit tests); new e2e proving
envelope-driven overflow folds + forwards; affected fixtures re-sized to sit
under honest accounting. typecheck + build clean; full suite green except one
pre-existing env-dependent failure unrelated to this change.
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 ework agent · qwen3.8-27b

[bot] 🏷 Review complete — code verified, tests reproduced. One item needs your confirmation before merge; one description nit.

Verdict: the fix is correct and complete. I checked out head 8e0a6ad (base a184ff9 = current master tip, so no rebase needed) and verified everything below locally.

What I validated

  • Bug is real, correctly layered. Master's preflightCompressIfNeeded sizes on estimateCoreMessages + images only; the billed input also carries system + tools, so in the band text < limit < text+overhead the trigger never fires → guaranteed upstream 400 on strict relays, and the post-fold gate could pass over-window billed input. Same class fix(#453): hard backstop for plugin-mode context overflow #467 fixed for the output clamp — this applies the same term to preflight. The layering is right (accounting fix, not a symptom patch).
  • All four consumption points covered: trigger, both post-fold gates, and the fold-loop floors via PreflightDeps.wireOverhead. That last one matters — without it the loop stops at "text fits" and then fail-fasts against the caller's overhead-aware gate (502 on foldable payloads). Good catch.
  • No double-counting. OpenAI: the kernel hoists the leading system/dev prefix out of CoreMessage[] (openaiToCore) and injectOpenaiSystem puts it back into the rebuilt body's messages → counted exactly once from the body scan. Anthropic/Responses: top-level system/instructions, not in messages. And max(lastInputTokens, payloadEstimate) is an alternative, not a sum — the measured baseline already includes overhead. One benign edge: a mid-conversation system/dev message on OpenAI would be counted twice (it stays in processedMessages AND matches the body scan) — conservative direction only (slightly earlier trigger), acceptable.
  • Both compression modes reasoned (per AGENTS.md §6): preflight runs unconditionally (no pluginMode gate), and estimateWireOverhead reads prepared.body post-injection, so proxy mode counts the ACP tools + compress prompt while plugin mode counts the client's own manifest (bili tools suppressed by design). README table correction is right.
  • Post-fold gate recompute is sound. Rebuilding via runPrepare() and gating on the actual forwarded payload eliminates the false-fit from noEmergencyTruncate trimming result.payloadEstimate more than normal prepare does. Reusing overheadEstimate/imageTokens across the rebuild is valid — system/tools/images are fold-invariant.
  • Pre-flight reproduced: npm run typecheck ✅ · npm run build ✅ (self-contained dist) · npm test 1071/1072 — the single failure is the env-dependent resolveClientCommand launcher test; I reproduced the identical failure on a master baseline worktree, confirming it's pre-existing as claimed (your count was 1070/1071, mine 1071/1072 — count drift, same single failure). Targeted run of wire-overhead, preflight-fail-fast, issue453-backstop, codex-compact-e2e, compaction-trigger-finality: all 32 pass, including the new track: plugin-mode proxy-side emergency truncation (last-resort backstop, deferred from #467) #470 e2e and the preflight fail-fast × soft-protection livelock: over-window payload with nothing foldable outside the protected zone 502s forever #330 relax e2e.
  • The bundled preflight fail-fast × soft-protection livelock: over-window payload with nothing foldable outside the protected zone 502s forever #330 relax code is coherent: noEmergencyTruncate (×100 window) keeps usage under truncate.threshold so the loop reaches the relax step; relax-retry is gated on the payload's own estimate (not the stale-floor currentTokens); exhausted detail is actionable ("Raise the model context window or restart the session").

⚠️ Needs your confirmation: this PR silently re-lands orphaned PR #337

GitHub shows #337 as merged (merged_at 2026-09-06T06:31:59Z, merge commit 6820e7d) — but that merge commit is unreachable from current master (a184ff9). Master contains none of #337's code (verified: no relaxedConfig / noEmergencyTruncate markers anywhere). This branch's history contains exactly that merge commit plus your fix, so merging #577 lands #330 soft-zone relax + #496 image-cost fix atomically with the #470 fix, even though GitHub displays #337 as already merged and the body only says "Closes #470".

The coupling is real (#470's e2e depends on the relax code; the early-exit restructuring interlocks with it), so bundling is probably intentional — but please confirm, and add a line to the PR body stating it re-lands #337 whose merge didn't stick to master, so future readers of master's history aren't confused.

Nit: stale "Test retuning" section

The description claims retunes of tests/codex-compact-e2e.test.ts (window 10k→12k) and tests/compaction-trigger-finality.test.ts (trimmed turn-1 filler) — neither file is in the diff (both unchanged from master), and both pass as-is. Looks like a leftover from an intermediate iteration (the PR head is wt-576-ework-daemon, not the branch name cited in the body). No action needed beyond correcting the description.

Pre-merge checklist item

server.ts's request pipeline changed, so per AGENTS.md §3 the gated codex e2e should run at least once against a live upstream before merge (CI manual dispatch with E2E_UPSTREAM_URL/E2E_UPSTREAM_KEY, or local sglang) — you noted the sandbox can't reach one. Everything else is green on my side; once you confirm the #337 bundling, this is ready for a human merge.

@ranxianglei

Copy link
Copy Markdown
Owner Author

Cross-ref: #576 implements the same #470 fix (same preflight wire-overhead root cause, opened ~1h earlier from the issue-thread analysis). Both are currently open against master — one of the two should be closed so #470 resolves through a single PR. #576 is mergeable against current master (53ce3ba) with pre-flight typecheck/test/build green (1067/1069 pass, 2 known plugin-agent env fails).

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 ework agent · qwen3.8-27b

[bot] 🏷 Dedup done: kept #577, closed #576 as duplicate (tagged duplicate, explained in its thread). Details below.

What I found

The two PRs are two independent implementations of the same fix, not one branch pushed twice:

#576 (closed) #577 (kept, this PR)
Head 8e0a6ad @ 2026-09-06_preflight-wire-overhead 3b746a5 @ wt-576-ework-daemon
Base orphaned #337 merge 6820e7d + fix single commit directly on current master a184ff9
Test retunes (codex-compact-e2e 10k→12k window, compaction-trigger-finality filler trim) claimed in description, absent from diff present in diff
Responses overhead accounting instructions only instructions plus developer items inside input

Note: my earlier review in this thread was of 8e0a6ad — which is actually #576's head (the issue body cited that branch name). The verification results carry over to this PR's head too, since I re-ran everything on 3b746a5:

  • npm run typecheck ✅ clean
  • npm test: 1071 tests, 1070 pass, 1 fail — the sole failure is the env-dependent resolveClientCommand launcher test, reproduced identically on a master baseline worktree (pre-existing, unrelated)
  • New e2e #470: text fits the window alone but system+tools push the billed input over → preflight folds and forwards passes; post-fold rebuild gate and openai hoisted-system handling match what I reviewed and endorsed

Why #577 wins

  1. Clean base = clean landing. Merging fix: count system+tools wire overhead in the preflight trigger (#470) #577 lands exactly the track: plugin-mode proxy-side emergency truncation (last-resort backstop, deferred from #467) #470 fix against real current master. Merging fix: count system+tools wire overhead in the preflight trigger (#470) #576 would have silently re-landed the rolled-back fix: preflight relaxes the soft-protected recent zone under overflow (#330) #337 work (preflight fail-fast × soft-protection livelock: over-window payload with nothing foldable outside the protected zone 502s forever #330 soft-zone relax + PR #489 后续: 默认图片计费 b64/4 在像素 tile 计费上游(官方 Anthropic/OpenAI)会硬 502 误杀多模态会话 #496 image-cost fix), which its body never discloses. Master was deliberately reset after fix: preflight relaxes the soft-protected recent zone under overflow (#330) #337's merge today; if fix: preflight relaxes the soft-protected recent zone under overflow (#330) #337 should land, it deserves its own disclosed PR (code intact at 6820e7d / branch 2026-08-29_preflight-soft-zone-relax).
  2. fix: count system+tools wire overhead in the preflight trigger (#470) #577 is the more complete iteration — it contains the test retunes fix: count system+tools wire overhead in the preflight trigger (#470) #576's description promised but didn't ship, plus slightly more thorough responses-protocol accounting.

Two notes

Everything else green on my side. Merge stays human-only per repo rules — ready when you are.

@ranxianglei ranxianglei closed this Sep 6, 2026
@ranxianglei

Copy link
Copy Markdown
Owner Author

Superseded by #576 (same #470 fix, merged). This branch also conflicts with current master. Closing as duplicate — thanks for the parallel implementation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

track: plugin-mode proxy-side emergency truncation (last-resort backstop, deferred from #467)

1 participant