Skip to content

fix(#767): image billing modes (pixels/bytes) end stale-baseline 502 loops on pixel-tile upstreams - #771

Merged
ranxianglei merged 3 commits into
masterfrom
2026-09-14_pixel-image-billing-767
Sep 14, 2026
Merged

ranxianglei merged 3 commits into
masterfrom
2026-09-14_pixel-image-billing-767

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

Fixes #767

Problem

Follow-up to #496/#500. On official Codex/OpenAI/Anthropic upstreams (pixel-tile billing), once a session's historical lastInputTokens baseline sits above the window (reported: 696789 vs a 272000 window), the default base64 length / 4 image cost closes both safe-forward paths at once:

So preflight fails 502 preflight_compress_failed ("Images alone account for ~N tokens") forever, even though the images bill only a few thousand tokens upstream (the issue's repro: byte estimate ~300K vs real billing 2458/1910 per image). Text-only sessions already self-heal via #300; only image-bearing payloads were bricked.

What changed

  • New imageBilling setting — global + per-provider providers.<url>.imageBilling ("auto" | "pixels" | "bytes"), env BILI_IMAGE_BILLING overrides both (live-read, no restart). Precedence: env > per-provider > global > auto host classification.
  • pixels mode: parses the container header (PNG/JPEG/WebP/GIF/BMP) with a bounded decode — never the full body — and charges OpenAI high-detail tile billing (512px tiles, short side scaled up to 768px, long side capped at 2048px → 85 + 170×tiles = 765–2805 tokens/image). Unparsable formats fall back to a flat 16384 — above both published first-party ceilings (OpenAI max 2805, Anthropic ~1534) yet far below base64/4 for any inline image over ~64KB.
  • bytes remains the default — conservative relay protection unchanged (多模态会话上下文控制失效 + preflight 救援 502(codex responses store:false) #488/PR #489 后续: 默认图片计费 b64/4 在像素 tile 计费上游(官方 Anthropic/OpenAI)会硬 502 误杀多模态会话 #496 guarantees intact); BILI_IMAGE_TOKEN_CAP still clamps on top of both modes; remote URLs stay flat 4096.
  • auto classifies by upstream host: openai.com / openai.azure.com / chatgpt.com / api.anthropic.com suffixes → pixels, everything else → bytes. An official Codex route (https://chatgpt.com/backend-api/codex) therefore gets tile billing with zero config.
  • Wired through all four estimate call sites (side-request guard, both output clamps, preflight gate) plus the forward() overflow-error path, and hot-reloaded with the rest of the config.

Recovery is one turn: usage reports replace lastInputTokens absolutely. Per the issue's request, #496's one-rejection-then-block guarantee for genuine byte-relay overflow is preserved — noOverflowEvidence is untouched; only the estimate layer changed.

Tests

New tests/issue767-pixel-billing.test.ts (9 tests):

  • unit: dimension parsing for all five container formats (incl. JPEG SOF behind EXIF padding), tile-math bounds, auto classification + explicit overrides + lookalike-host rejection, cap interplay, remote URL cost;
  • e2e (mock Responses upstream mirroring the issue's repro): stale 696789 baseline + two large base64 screenshots → 200 under pixels, both images forwarded verbatim, baseline recovers below the window, steady state continues; identical payload under default bytes billing → 502 fail-fast, zero images forwarded (relay protection pinned); learned-limit-only variant closes forward-once too; env precedence check.

Pre-flight: npm run typecheck ✓ · npm test 1438 pass / 0 fail (2 default-gated E2E skips) ✓ · npm run build ✓

Docs: CONFIGURATION.md + CONFIGURATION.zh-CN.md (global & per-provider imageBilling, BILI_IMAGE_BILLING, refreshed BILI_IMAGE_TOKEN_CAP row), CHANGELOG entry.

For the reporter

Once released, the process-wide BILI_IMAGE_TOKEN_CAP=16384 workaround can be dropped — the official Codex route bills images by tiles automatically (auto). Keep explicit control if desired: "imageBilling": "pixels" on that provider, or Environment=BILI_IMAGE_BILLING=pixels.

…loops on pixel-tile upstreams

On official Codex/OpenAI/Anthropic upstreams (pixel-tile billing), once a session's historical lastInputTokens baseline sits above the window, the default base64/4 image cost closes BOTH safe-forward paths at once (#496 forward-once requires a sub-window baseline; #300 stale-baseline fit requires payloadEstimate < limit) — preflight then fails 502 forever even though the images bill only a few thousand tokens upstream.

- new global + per-provider imageBilling ("auto"|"pixels"|"bytes"), env BILI_IMAGE_BILLING overrides both (live-read); auto classifies first-party pixel hosts (openai.com / openai.azure.com / chatgpt.com / api.anthropic.com)

- pixels mode parses the container header (PNG/JPEG/WebP/GIF/BMP, bounded decode — never the full body) and charges OpenAI high-detail tile billing (85 + 170x tiles = 765..2805 per image); unparsable formats fall back to a flat 16384

- bytes stays the conservative default for byte-counting relays; BILI_IMAGE_TOKEN_CAP still clamps both modes; remote URLs stay flat 4096

- regression tests: recovery e2e (stale 696789 baseline + two large screenshots forwards and recovers under pixels), bytes-mode control (same payload still fails fast, zero images forwarded), learned-limit-only variant, env precedence; typecheck + full suite (1438 pass) + build green
…RL at all gates

imageBillingFor received origin-only URLs at four of five call sites, so a per-provider imageBilling override under a path-qualified provider key (the pattern CONFIGURATION.md documents) was only honored by the forward() overflow gate while preflight/clamps/guard silently fell back to auto classification. Route lookups now use route.rewrittenUrl like every other per-route setting (passthrough/compress/proxy). New e2e regression pin: path-qualified key + stale baseline recovers under explicit pixels; verified to 502 without the fix.
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 Review of PR #771 (fix for #767) — one real wiring gap found, fixed directly on the branch as 3459c59; everything else checks out.

Verification (all run on the branch)

  • npm run typecheck ✓ · npm test 1439 pass / 0 fail (2 default-gated E2E skips) ✓ · npm run build ✓ (after my fix commit; original claim of 1438/0 reproduced exactly before it)
  • Diff cleanliness: single commit on current master, 7 files (+690/−24), all on-topic, no version bump, no lockfile churn ✓
  • Duplicate screen: no other open/closed issue covers this; source issue [Bug] #496 后续:旧 baseline 超窗时官方 Codex 图片估算持续误触发 502 #767 properly referenced ✓
  • Tile math independently re-derived: 85 + 170×tiles, short side scaled up to 768px, long side capped 2048px → [765, 2805]; test vectors match OpenAI's published high-detail model ✓
  • Header parsers: bounded decode only (64 b64 chars non-JPEG / ~262KB JPEG SOF scan), WebP VP8/VP8L/VP8X layouts verified against RFC 6386 bit layout, JPEG skips RST/D8/01 and breaks on malformed segment length or SOS-without-SOF ✓. Fallback 16384 sits above both first-party ceilings (OpenAI 2805 / Anthropic ~1534) yet far below base64/4 for any inline image over ~64KB ✓
  • Relay protection pinned: bytes stays the default everywhere (incl. signature defaults), noOverflowEvidence untouched per [Bug] #496 后续:旧 baseline 超窗时官方 Codex 图片估算持续误触发 502 #767's explicit request, cap still clamps both modes, remote URLs stay flat 4096 ✓
  • Both compression modes considered: the change is purely in the estimate layer shared by proxy/plugin modes — no wire-format change, so no mode-divergence risk ([Bug] ACP: ≥2 compressed blocks render as multiple system messages → SGLang (vLLM-compatible) upstreams return 400 #377 class) ✓
  • Hot reload: handleConfigReload picks up global imageBilling; env read live per request ✓; suffix matching is lookalike-safe (host === h || endsWith("." + h)) ✓
  • Docs accurate in EN+ZH; web UI verified too — parseRouteEntry preserves imageBilling, so per-route values round-trip through the panel's providers PUT (top-level key is file/env-only by design, which the docs state) ✓

Fixed on the branch: per-route override invisible at 4 of 5 gates for path-qualified provider keys

imageBillingFor() received origin-only URLs at four call sites (src/server.ts:1456 side-request guard, :2323/:2558 output clamps, :3085 preflight gate) while only the forward() overflow path (:3649) got the full URL. Since findRoute does longest-prefix matching, a per-provider imageBilling under a path-qualified key — e.g. the exact pattern your own CONFIGURATION.md documents ("https://chatgpt.com/backend-api/codex": { "imageBilling": "pixels" }) — was never found at those four sites and silently fell back to auto classification. Concretely: an explicit "bytes" safety override on such a route would not apply at the preflight/clamp/guard layer (the looser pixels estimate would be used instead) even though the user believes it's active. Every other per-route setting (passthrough :1199, compress :1596, proxy :2956) resolves against route.rewrittenUrl — full URL — so this was also an inconsistency with established convention.

Fix (3459c59, 2 files, +48/−8): all five sites now look the route up with the full upstream URL (route?.rewrittenUrl, threaded through prepareOpenai/prepareResponses via a trailing optional param — upstreamOrigin itself keeps its origin semantics for the deepseek regex / compressProtocol / session meta). Origin-keyed routes still match (prefix semantics unchanged), MITM traffic follows the same mitm:// key distinction as every sibling setting, and auto classification is unaffected when no route matches. Added an e2e regression pin (path-qualified key + stale 696789 baseline recovers under explicit pixels) and verified it fails with 502 on the pre-fix code. Post-fix suite: 1439/0.

Noted, not blocking (pre-existing, left untouched)

resolveCompressProtocol(opts.routes, upstreamOrigin) at src/server.ts:2431 has the same latent origin-only lookup shape for compressProtocol under path-qualified keys. Pre-existing behavior, different setting, out of scope here — worth a follow-up issue if you want that convention unified.

Merge is yours — I can't merge PRs.


中文摘要: 修复了按路由 imageBilling 覆盖在路径限定 provider key(即文档示例的写法)下只在 forward() 溢出门生效、其余四个门(preflight/两个 clamp/side-request guard)静默回退 auto 的问题——改为与 passthrough/compress/proxy 一致地用完整上游 URL 查路由,并补了回归测试(已验证旧代码下该测试 502 失败);其余审查项(tile 计费数学、五种容器头解析、relay 保护不变量、双压缩模式、热重载、双语文档)均通过,可以合并。

@ranxianglei

Copy link
Copy Markdown
Owner Author

冲突

# Conflicts:
#	CHANGELOG.md
#	src/config.ts
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 冲突已解决并推送到 PR 分支(merge commit fac0144)。

master 自分支点以来前进了不少(kimi/jcode launcher、reasoning guard #739、session 加密 #708/#711、release v0.1.109,acp-kernel → 0.0.69),与 #771 的冲突只有两处,都是"同一位置各自加了不同内容":

  • src/config.ts — import 行:本分支加 ImageBillingMode、master 加 ReasoningGuardConfig → 两个都保留。
  • CHANGELOG.md — Features 段顶部:本分支的 image-billing 条目 vs master 的 session-encryption 条目 → 两条都保留。

src/server.ts 自动合并干净;已逐一核对五个 billing 调用点(side-request guard / 两个 output clamp / preflight / forward overflow)和 config.ts 的 imageBilling 字段在合并后全部完好,reasoningGuard 的新代码与本 PR 无交互。

合并后全量验证(deps 已按新 lockfile 重装):npm run typecheck ✓ · npm test 1477 pass / 0 fail(2 default-gated E2E skips)✓ · npm run build ✓。PR 现在应显示可合并状态,merge 仍归你操作。


中文摘要: 把最新 master 合入 PR 分支解决了两处冲突(config.ts 的 import 行与 CHANGELOG Features 顶部条目,均为双方各自新增、全部保留),验证了 image-billing 改动在合并后完整无损且 typecheck/测试/构建全绿,已推送,可以合并。

@github-actions

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-09-14_pixel-image-billing-767 (fac0144)

Option A — Install from npm PR tag (recommended)

npm install -g billion-context@pr-771

Each push to this PR publishes a new version under the pr-771 npm tag.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr771.tgz
npm install -g package

This comment is automatically updated on each push.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] #496 后续:旧 baseline 超窗时官方 Codex 图片估算持续误触发 502

1 participant