Skip to content

fix: forward-once-then-learn for image-dominated payloads (#496) - #500

Merged
ranxianglei merged 2 commits into
masterfrom
2026-09-03_issue-496-forward-once-learn
Sep 4, 2026
Merged

ranxianglei merged 2 commits into
masterfrom
2026-09-03_issue-496-forward-once-learn

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

What

Follow-up to #496 — fixes the residual design risk from PR #489's image-token accounting. The default per-image estimate (base64/4, uncapped) matches byte-billing relays but overestimates pixel-tile upstreams (official Anthropic/OpenAI) by up to ~200x, so a session whose estimated image floor alone exceeded the window was hard-failed 502 preflight_compress_failed even though the real cost was a few K tokens — a regression vs master for official-API multimodal users (e.g. bili claude pasting screenshots).

Fix — forward-once-then-learn (src/server.ts)

When images are the sole over-window component (textEstimate < limit) and there is no upstream overflow evidence yet (session.stats.lastInputTokens < limit and no learned limit for this model), the fit gate forwards once instead of hard-failing, letting the upstream arbitrate billing:

  • Pixel-tile upstreams (official Anthropic/OpenAI): accept → usage reports small → zero false positives, zero behavior change.
  • Byte-billing relays (多模态会话上下文控制失效 + preflight 救援 502(codex responses store:false) #488): reject once (400) → the existing self-heal learns the true window (it already counts rejected image tokens) → every later request fails fast with an accurate message. Exactly one rejected forward — strictly better than master's infinite 400 loop, no new knob.

With either evidence signal present we trust the estimate and fall through to fold / fail-fast exactly as before.

Scope note — stacked on PR #489

This branch is based on f7deccc (PR #489 tip), so it carries #489's content into master alongside this fix. The second commit also carries #489's doc gaps noted in #496 items 1 & 2 (BILI_IMAGE_TOKEN_CAP env rows + [Unreleased] changelog entries for #488 and #496). If those were already added to #489, drop the doc commit.

Pre-flight

  • npm run typecheck — clean
  • npm test — 986 pass / 1 fail / 987 total. The 1 failure (launcher.test.ts → resolveClientCommand: codex/claude resolve to themselves) is a pre-existing environmental failure, reproduced identically on pristine base f7deccc; unrelated to this change.
  • Relevant tests: tests/fix-488-multimodal.test.ts (多模态会话上下文控制失效 + preflight 救援 502(codex responses store:false) #488-A rewritten → tile upstream accepts the once-forwarded payload) + new tests/issue496-forward-once-learn.test.ts (byte relay: one rejected forward then fail-fast) — 8/8 pass.
  • npm run build — success

Not addressed here (tracked in #496, non-blocking)

ework-agent added 2 commits September 3, 2026 22:41
The default per-image estimate (base64/4, uncapped) matches byte-billing relays (#488) but overestimates pixel-tile upstreams (official Anthropic/OpenAI) by up to ~200x, so a session whose estimated image floor alone exceeded the window was hard-failed 502 preflight_compress_failed even though real cost was a few K tokens - a regression vs master for official-API multimodal users.

When images are the sole over-window component (textEstimate < limit) and there is no upstream overflow evidence yet (lastInputTokens < limit and no learned limit for the model), forward once so the upstream arbitrates billing: tile upstreams accept (zero behavior change); byte relays reject once, self-heal learns the window, later requests fail-fast. Exactly one rejected forward - #488's 400 loop stays broken, no new knob.
Carrying PR #489's doc gaps noted in #496 (items 1 & 2): add BILI_IMAGE_TOKEN_CAP to the CONFIGURATION env tables (EN+ZH) and add [Unreleased] changelog entries for the #488 image-count fix and the #496 forward-once fix. Drop these hunks if already added to #489.
@github-actions

github-actions Bot commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-09-03_issue-496-forward-once-learn (431ff90)

Option A — Install from npm PR tag (recommended)

npm install -g billion-context@pr-500

Each push to this PR publishes a new version under the pr-500 npm tag.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr500.tgz
npm install -g package

This comment is automatically updated on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 ework agent · qwen3.8-27b

[bot] 🏷 Reviewed — verified independently. LGTM, ready for a human merge (I won't merge per policy).

I checked out the head (431ff90) in a clean tree and re-ran every pre-flight claim rather than taking them on faith. All hold:

Check Result
npm run typecheck clean
npm run build success
Full suite (tests/*.test.ts) 986 pass / 1 fail / 987 total — matches your report
Targeted (issue496-forward-once-learn + fix-488-multimodal) 8/8 pass, incl. both decisive scenarios (tile upstream accepts the once-forwarded payload; byte relay = one rejected forward then fail-fast)

The single failure, launcher.test.ts → resolveClientCommand: codex/claude resolve to themselves, is confirmed pre-existing/environmental: it reproduces identically on pristine base f7deccc, and this PR touches no launcher/client-config code. Not a regression.

Design — the mechanism actually holds

Traced preflightCompressIfNeeded (src/server.ts) end-to-end:

  • Gate: imageTokens > 0 && textEstimate < limit && noOverflowEvidence, where noOverflowEvidence = lastInputTokens < limit && learnedLimit === undefined. The textEstimate < limit guard correctly excludes text-dominant overflow (which should attempt compression, not blind-forward) — good call.
  • Byte-relay "exactly one rejected forward" hinges on the self-heal reliably learning from the rejection. Verified it writes a learned limit in both sub-cases in forward(): parseable window via info.window, and the unparseable case (context_window_exceeded) via payloadEstimate = estimateCoreMessages + rejectedImageTokens (the 多模态会话上下文控制失效 + preflight 救援 502(codex responses store:false) #488 image accounting). So request N+1 sees learnedLimit !== undefined → skips forward-once → fail-fast. No reintroduction of the 多模态会话上下文控制失效 + preflight 救援 502(codex responses store:false) #488 loop. ✓
  • §6 both compression modes: preflight is invoked unconditionally at src/server.ts:1294 for plugin and proxy mode across all three wires. The change only alters the forward-vs-fail decision, never the wire bytes, so it can't break one mode while fixing the other. ✓

Scope note — now moot, but confirms the merge is safe

Your note said the branch "carries #489's content into master." That's no longer accurate: #489 is already merged to master (current tip 3eb885f). So a trial merge of this PR into real master comes back clean, zero conflicts, with a net delta of exactly the intended 6 files (+198/−10) — no double-application of #489. On the doc commit: I checked master and it has neither the BILI_IMAGE_TOKEN_CAP rows nor the #488/#496 changelog entries, so those gaps are real — keep the docs commit (all 6 files). Also confirmed BILI_IMAGE_TOKEN_CAP is genuinely wired (src/image-tokens.ts:14-21, applyCap = Math.min(cost, cap)), so the env-table row isn't aspirational.

Minor, non-blocking

  1. The "one rejected forward" guarantee depends on inspectContextOverflow() recognizing the relay's error as an overflow. A relay whose rejection format isn't classified would allow a second forward before fail-fast. This is not a new risk — it's the same coupling master/fix: multimodal token accounting + store:false in Responses preflight summary (#488) #489 already relies on for its own fail-fast path — just flagging the dependency.
  2. Cosmetic: the 多模态会话上下文控制失效 + preflight 救援 502(codex responses store:false) #488 changelog entry ships under [Unreleased] here even though 多模态会话上下文控制失效 + preflight 救援 502(codex responses store:false) #488 already went out in v0.1.77 without one, so it'll surface in the next release's notes rather than retroactively. Harmless.

Nothing blocking. Recommend you merge when ready.

@ranxianglei

Copy link
Copy Markdown
Owner Author

Independent review (second session) — no functional bug found; LGTM.

Verified on top of current master (post-#504/#505/#506, i.e. with weak-overflow learning now live):

  • No loop amplification: noOverflowEvidence requires lastInputTokens < limit && learnedLimit === undefined. After one byte-relay 400 the strong-signal path writes learnedContextLimits[model] → subsequent requests fail-fast (pinned by the e2e test: exactly ONE forward across 3 requests). After a tile upstream ACCEPT, real usage lands well under the window and stays eligible for forward-once — correct, since the estimate is provably wrong for that upstream and each forward is a served request, not a retry.
  • No interaction bug with fix: #498 weak-overflow learning + #499 P1a affinity persistence + P1b cache-collapse warn #504 weak-overflow: weak signals only arm at input/window ≥ 90% with input from real usage — a tile-billed accept reports small real usage, so it never poisons noOverflowEvidence.
  • Lookup symmetry: learnedMap?.[model] ?? learnedContextLimit matches both write sites (strong path and weak-overflow write model-scoped when model is known, scalar otherwise).
  • Placement: the early return prepared correctly precedes the fold path, the fix: honor anthropic-beta context-1m to size the context window #301 stale-baseline forward, and the nudge block — no double-forward, no skipped nudge bookkeeping.
  • Combined-tree pre-flight re-run here: both this and fix: stop legacy-id migration destroying anonymous pfa sessions (#499 prerequisite) #502 merged onto current master → typecheck ✅, 1020/1022 (2 known env fails).

From my side this is merge-ready as-is.

@ranxianglei
ranxianglei merged commit bb855cc into master Sep 4, 2026
6 checks passed
ranxianglei pushed a commit that referenced this pull request Oct 3, 2026
…eady path (#1960)

The steady request path (client echoes the round back) silently dropped
redacted_thinking blocks because anthropicToCore had no case for them, so
every later request rebuilt the latest assistant message without the block
and Anthropic answered 400 "cannot be modified" forever — the half of #1960
that kept the session stuck after the compress turn.

- acp-kernel pin 0.0.99 → 0.0.101 (exact), which carries kernel PR #500
  (redacted_thinking sidecar round-trip). Also rides along from v0.0.100+:
  #483 ref capacity widened to 9,999,999, #484 case-insensitive protected
  tool matching, #478 nudge cadence re-anchor, #493 language-preservation
  behind an opt-in flag (default off), and #499 prune/recommend shared
  per-turn index (hundreds of ms saved on ordinary requests in large
  sessions — release-note material).
- tests/issue1960-steady-redacted.test.ts: client-echoed
  [thinking(sig), redacted(data), tool_use] must survive prepare→outbound
  byte-identical in order. Verified failing on 0.0.99, passing on 0.0.101.

Preflight: typecheck clean; targeted loop/thinking/reasoning suites 186/186
green incl. both new #1960 files; build OK.
@ranxianglei
ranxianglei deleted the 2026-09-03_issue-496-forward-once-learn branch October 15, 2026 05:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant