diff --git a/CHANGELOG.md b/CHANGELOG.md index 73aba82fef..06d891eb9f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,6 +19,7 @@ Versions follow the merge of a `*_release-v*` branch; CI publishes to npm on tag ### Features +- **Image billing modes (`imageBilling` / `BILI_IMAGE_BILLING`): pixel-tile upstreams no longer get permanently 502'd by byte-based image estimates (#767)**: follow-up to #496/#500 — on official Codex/OpenAI/Anthropic upstreams (pixel-tile billing), once a session's historical `lastInputTokens` baseline sits above the window, the default `base64 length / 4` image cost closes BOTH safe-forward paths at once (#496 forward-once requires a sub-window baseline; #300 stale-baseline fit requires `payloadEstimate < limit`), so preflight fails 502 `preflight_compress_failed` ("Images alone account for ~N tokens") forever even though the images bill only a few thousand tokens upstream (repro: two screenshots, byte estimate ~300K vs real billing ~4.4K). New global + per-provider `imageBilling` (`"auto" | "pixels" | "bytes"`; env `BILI_IMAGE_BILLING` overrides both, live-read per request): `pixels` parses the image header (PNG/JPEG/WebP/GIF/BMP, bounded decode — never the full body) and charges first-party high-detail tile billing (512px tiles, short side scaled up to 768px, long side capped at 2048px → 765–2805 tokens per image; unparsable formats fall back to a flat 16384); `bytes` keeps the conservative default so byte-counting relays retain their #488/#496 protection unchanged. `auto` classifies by upstream host (`openai.com` / `openai.azure.com` / `chatgpt.com` / `api.anthropic.com` suffixes → pixels; everything else → bytes). Remote URLs stay flat 4096 in both modes; `BILI_IMAGE_TOKEN_CAP` still clamps on top of both. A stale baseline now self-heals within one turn of pixel-billed traffic (usage reports replace the baseline absolutely), while #496's one-rejection-then-block guarantee for genuine byte-relay overflow is preserved — pinned by regression tests: recovery e2e (stale 696789 baseline + two large screenshots forwards and recovers under `pixels`), bytes-mode control (same payload still fails fast with zero images forwarded), learned-limit-only variant, and env-precedence check. - **Session-file encryption at rest behind `BILI_ENCRYPTION_KEY` (#708, PR #711)**: for deployments on untrusted nodes, persisted session files (block summaries plus up to ~16k tokens of folded conversation per session — effectively full conversation content) can now be stored as `BILIENC1` AES-256-GCM over zstd-compressed JSON (zstd via built-in node:zlib on Node ≥ 22.15, raw mode byte on older runtimes; compression also shrinks files ~5–10×). The key is env-only — deliberately NO key-file support, since a key file would sit on the same untrusted filesystem as the data — exactly 32 bytes hex or base64, never written to disk or logs; an invalid value aborts startup (fail fast instead of silently running unencrypted). When enabled, boot takes over legacy plaintext files once: each is re-encoded and atomically replaced in place (the rename IS the old-file deletion; a crash mid-migration self-heals on the next boot), covering `.fb.json` spill files too; unreadable files are left in place with a warning. Wrong key at boot → GCM auth failure → affected sessions skip as corrupt (logged, no crash). Mechanism is the kernel's optional `StateStore` codec hook (acp-kernel#249 → 0.0.64); unset key = zero behavior change. - **Per-route `passthrough` option for providers (#661, PR #662)**: `"providers": { "mitm://zcode.z.ai": { "passthrough": true } }` forwards matching requests byte-for-byte — no kernel round-trip, no render tags, no `prompt_cache_key` removal, response piped verbatim, no session state — the scoped sibling of the global `passthrough` switch. `mitm://` keys target only MITM (login-client) traffic; plain `https://` host keys cover both MITM and `/bili/` traffic. diff --git a/CONFIGURATION.md b/CONFIGURATION.md index bbecb8feee..ece697512a 100644 --- a/CONFIGURATION.md +++ b/CONFIGURATION.md @@ -122,11 +122,18 @@ Top-level keys that control how the proxy listens and behaves globally. - **Status:** ACTIVE - **Description:** Upstream HTTP proxy (`http://host:port`) used for the proxy's **own** outbound connections to model providers. SOCKS5 is not supported. A per-URL `proxy` set inside a `providers` entry overrides this for that provider. An empty string means "explicitly direct" — it disables any environment/system proxy fallback for all providers. +### `imageBilling` + +- **Type:** `"auto" | "pixels" | "bytes"` +- **Default:** `"auto"` +- **Status:** ACTIVE +- **Description:** How inline (base64) images are charged by the preflight size gate and output clamp (#488/#496/#767). `"bytes"` charges each image at `base64 length / 4` tokens — conservative, and correct for byte-billing relays. `"pixels"` parses the image header (PNG/JPEG/WebP/GIF/BMP) without decoding the body and charges first-party pixel-tile billing (OpenAI high-detail model: 512px tiles, short side scaled up to 768px, long side capped at 2048px → 765–2805 tokens per image; unparsable formats fall back to a flat 16384). Remote (`https://`) images always charge a flat 4096 in either mode. A per-provider `providers..imageBilling` wins over this global, and the `BILI_IMAGE_BILLING` env var wins over both (live-read, no restart). + --- ## Providers -The `providers` block maps **upstream URLs** to per-provider configuration. Each key is a URL prefix; each value can declare model context windows, a per-provider proxy, a compression protocol, compression overrides, and a per-route passthrough. +The `providers` block maps **upstream URLs** to per-provider configuration. Each key is a URL prefix; each value can declare model context windows, a per-provider proxy, a compression protocol, compression overrides, an image billing mode, and a per-route passthrough. ```jsonc { @@ -201,6 +208,21 @@ A shallow key (`https://open.bigmodel.cn`) matches every path on that host. A de } ``` +### `imageBilling` + +- **Type:** `"auto" | "pixels" | "bytes"` +- **Default:** *(global `imageBilling`, then `"auto"`)* +- **Status:** ACTIVE +- **Description:** Per-route override of how inline images are charged by the size gate (#767). Set `"pixels"` for official Codex/OpenAI/Anthropic endpoints (pixel-tile billing) and keep `"bytes"` for byte-counting relays — under byte billing, a stale over-window baseline plus large base64 screenshots fails preflight with 502 forever even though the images bill only a few thousand tokens upstream. When unset at both levels, billing is auto-selected from the upstream host: hosts ending in `openai.com`, `openai.azure.com`, `chatgpt.com`, or `api.anthropic.com` → `pixels`; everything else → `bytes`. The `BILI_IMAGE_BILLING` env var overrides both config levels: + + ```jsonc + { + "providers": { + "https://chatgpt.com/backend-api/codex": { "imageBilling": "pixels" } + } + } + ``` + --- ## Compression Tuning @@ -438,7 +460,8 @@ Environment variables take precedence over the config file. They are useful for | `ACP_COMPRESS_TOOL` | Set to `0` to disable tool injection (same as `"compress.injectTool": false`). | | `ACP_COMPRESS_NUDGE` | Set to `0` to disable nudge injection (same as `"compress.injectNudge": false`). | | `ACP_MODEL_CONTEXT_LIMIT` | Override the context limit globally (absolute token count). | -| `BILI_IMAGE_TOKEN_CAP` | Cap the per-image token estimate used by the preflight size gate and output clamp (#488/#496). By default an inline `data:` image counts as `base64 length / 4` tokens with **no cap** — correct for byte-billing relays, but a large over-estimate for pixel-tile upstreams (official Anthropic/OpenAI), which bill each image at roughly 1.1K–1.6K tokens regardless of byte size. Set this to your upstream's per-image tile cost so the gate reflects real billing; unset = no cap (current default). | +| `BILI_IMAGE_TOKEN_CAP` | Cap the per-image token estimate used by the preflight size gate and output clamp (#488/#496). By default an inline `data:` image counts as `base64 length / 4` tokens with **no cap** — correct for byte-billing relays, but a large over-estimate for pixel-tile upstreams (official Anthropic/OpenAI). For pixel-tile upstreams prefer [`imageBilling`](#imagebilling) (`"pixels"`, or `BILI_IMAGE_BILLING=pixels`) which charges real tile billing instead of capping the byte estimate; the cap still applies on top of both billing modes as a blanket ceiling. Unset = no cap (default). | +| `BILI_IMAGE_BILLING` | Override the image billing mode used by the preflight size gate and output clamp (#767): `pixels` or `bytes`. Live-read per request (no restart); beats the global `imageBilling` and every per-provider `providers..imageBilling`. Use `bytes` to force conservative billing on a route configured `"pixels"` (e.g. a byte-counting relay behind an OpenAI lookalike host), or `pixels` to enable tile billing process-wide without editing config. See [`imageBilling`](#imagebilling). | | `BILI_PREFLIGHT_HOLD_MS` | Grace period (ms) before a long preflight compression starts holding the client with keep-alive bytes (default `30000`; see #568 / README "Preflight hold"). | | `BILI_CONFIG_FILE` | Override the config file path (point at any JSON file). | | `ACP_PORT` / `PORT` | Override the listen port. | diff --git a/CONFIGURATION.zh-CN.md b/CONFIGURATION.zh-CN.md index fcfcbd6849..a28f31e0bd 100644 --- a/CONFIGURATION.zh-CN.md +++ b/CONFIGURATION.zh-CN.md @@ -122,11 +122,18 @@ - **状态:** ACTIVE - **说明:** 用于代理**自身**到模型 provider 的出站连接的上游 HTTP 代理(`http://host:port`)。不支持 SOCKS5。在 `providers` 条目内设置的按 URL 的 `proxy` 会针对该 provider 覆盖此项。空字符串表示"显式直连" —— 为所有 provider 禁用任何环境/系统代理回退。 +### `imageBilling` + +- **类型:** `"auto" | "pixels" | "bytes"` +- **默认值:** `"auto"` +- **状态:** ACTIVE +- **说明:** 预检尺寸门与输出钳制对内联(base64)图片的计费方式(#488/#496/#767)。`"bytes"` 按 `base64 长度 / 4` 计 token —— 保守,且对字节计费 relay 正确。`"pixels"` 只解析图片头(PNG/JPEG/WebP/GIF/BMP)、不解码完整图像,按第一方像素 tile 计费(OpenAI high-detail 模型:512px tile、短边放大到 768px、长边封顶 2048px → 每图 765–2805 token;无法解析的格式回退为固定 16384)。远程(`https://`)图片在两种模式下都固定计 4096。按 provider 的 `providers..imageBilling` 优先于本全局项,而 `BILI_IMAGE_BILLING` 环境变量优先于两者(实时读取,无需重启)。 + --- ## Providers -`providers` 块将**上游 URL** 映射到按 provider 的配置。每个键是一个 URL 前缀;每个值可以声明模型上下文窗口、按 provider 的代理、压缩协议、压缩覆盖项,以及按路由的透传开关。 +`providers` 块将**上游 URL** 映射到按 provider 的配置。每个键是一个 URL 前缀;每个值可以声明模型上下文窗口、按 provider 的代理、压缩协议、压缩覆盖项、图片计费模式,以及按路由的透传开关。 ```jsonc { @@ -199,6 +206,21 @@ } ``` +### `imageBilling` + +- **类型:** `"auto" | "pixels" | "bytes"` +- **默认值:** *(全局 `imageBilling`,再回退 `"auto"`)* +- **状态:** ACTIVE +- **说明:** 按路由覆盖尺寸门的图片计费方式(#767)。官方 Codex/OpenAI/Anthropic 端点(像素 tile 计费)设 `"pixels"`,字节计数 relay 保持 `"bytes"` —— 字节计费下,历史 baseline 超窗加上大 base64 截图会让 preflight 永远 502,而上游实际每图只收几千 token。两级都未显式设置时,按上游 host 自动选择:以 `openai.com`、`openai.azure.com`、`chatgpt.com`、`api.anthropic.com` 结尾的 host → `pixels`,其余 → `bytes`。`BILI_IMAGE_BILLING` 环境变量覆盖两级配置: + + ```jsonc + { + "providers": { + "https://chatgpt.com/backend-api/codex": { "imageBilling": "pixels" } + } + } + ``` + --- ## 压缩调优 @@ -436,7 +458,8 @@ | `ACP_COMPRESS_TOOL` | 设为 `0` 禁用工具注入(等同 `"compress.injectTool": false`)。 | | `ACP_COMPRESS_NUDGE` | 设为 `0` 禁用 nudge 注入(等同 `"compress.injectNudge": false`)。 | | `ACP_MODEL_CONTEXT_LIMIT` | 全局覆盖上下文上限(绝对 token 数)。 | -| `BILI_IMAGE_TOKEN_CAP` | 预检尺寸门与输出钳制用的单图 token 估算上限(#488/#496)。默认内联 `data:` 图片按 `base64 长度 / 4` 计 token、**无上限** —— 对字节计费 relay 正确,但对像素 tile 计费的官方上游(Anthropic/OpenAI)会严重高估(后者无论字节多少,每图约计 1.1K–1.6K token)。设为你上游的单图 tile 成本,可让尺寸门反映真实计费;不设置 = 无上限(当前默认)。 | +| `BILI_IMAGE_TOKEN_CAP` | 预检尺寸门与输出钳制用的单图 token 估算上限(#488/#496)。默认内联 `data:` 图片按 `base64 长度 / 4` 计 token、**无上限** —— 对字节计费 relay 正确,但对像素 tile 计费的官方上游(Anthropic/OpenAI)会严重高估(后者无论字节多少,每图约计 1.1K–1.6K token)。像素 tile 上游建议改用 [`imageBilling`](#imagebilling)(`"pixels"`,或 `BILI_IMAGE_BILLING=pixels`),按真实 tile 计费而非截断字节估算;该上限仍在两种计费模式之上作为统一天花板生效。不设置 = 无上限(默认)。 | +| `BILI_IMAGE_BILLING` | 覆盖预检尺寸门与输出钳制的图片计费模式(#767):`pixels` 或 `bytes`。每次请求实时读取(无需重启);优先于全局 `imageBilling` 与所有按 provider 的 `providers..imageBilling`。在配置为 `"pixels"` 的路由上强制保守计费用 `bytes`(例如 OpenAI 同形 host 后面的字节计数 relay),或不想改配置文件就全进程启用 tile 计费用 `pixels`。详见 [`imageBilling`](#imagebilling)。 | | `BILI_PREFLIGHT_HOLD_MS` | 预压缩超过该宽限期(毫秒)后,代理提前提交响应并用保活字节挂住客户端(默认 `30000`;见 #568 / README「预压缩挂起」)。 | | `BILI_CONFIG_FILE` | 覆盖配置文件路径(指向任意 JSON 文件)。 | | `ACP_PORT` / `PORT` | 覆盖监听端口。 | diff --git a/src/config.ts b/src/config.ts index 142aad99ac..972a362085 100644 --- a/src/config.ts +++ b/src/config.ts @@ -6,6 +6,7 @@ import { log as loggerLog } from "./logger.js"; import { validateHttpProxy, type ProxyFallbackOptions } from "./upstream-proxy.js"; import { parseCompatRoles } from "./compat-roles.js"; +import type { ImageBillingMode } from "./image-tokens.js"; import type { ReasoningGuardConfig } from "./reasoning-guard.js"; export function safeReadJson(path: string): unknown { @@ -57,6 +58,12 @@ export type ProviderRoute = { * verbatim, no session state. For upstreams whose anti-cheat fingerprints * the request body (e.g. ZCode 405/3012). */ passthrough?: boolean; + /** Per-provider image billing mode (#767): "bytes" = ceil(base64/4) + * (conservative, matches byte-counting relays); "pixels" = dimension- + * based tile estimate (matches first-party pixel-tile upstreams); + * "auto" (default) classifies known first-party pixel hosts. Wins over + * the global `imageBilling`; env BILI_IMAGE_BILLING wins over both. */ + imageBilling?: ImageBillingMode; }; export type ProviderRoutes = Record; // key = upstream URL prefix (the /bili/ string) @@ -322,6 +329,9 @@ export type ProxyOptions = { * forwarded body for upstreams without the developer role (#552). Empty = * byte-for-byte transparent. */ compat: { roles: Record }; + /** Global-level image billing mode (#767); per-provider route entries + * override it, env BILI_IMAGE_BILLING overrides both. undefined = auto. */ + imageBilling?: ImageBillingMode; sessionHeader: string; log: boolean; debug: boolean; @@ -464,6 +474,7 @@ export function loadOptions(env: NodeJS.ProcessEnv = process.env): ProxyOptions routing: parsePromptCacheRouting(env.ACP_PROMPT_CACHE_ROUTING ?? fileConfig.promptCache?.routing), }, compat: { roles: parseCompatRoles(fileConfig.compat?.roles) ?? {} }, + imageBilling: parseImageBilling(fileConfig.imageBilling), sessionHeader: env.ACP_SESSION_HEADER ?? fileConfig.sessionHeader ?? "x-acp-session", log: env.ACP_LOG !== "0" && fileConfig.log !== false, debug: (env.ACP_DEBUG ?? (fileConfig.debug ? "1" : "0")) === "1", @@ -519,6 +530,10 @@ type FileConfig = { * upstreams accept (e.g. `{"developer":"system"}`) — applied to the * final forwarded body for openai/responses requests (#552). */ compat?: { roles?: Record }; + /** Global image billing mode (#767): "auto" | "pixels" | "bytes". + * Per-provider `imageBilling` overrides it; env BILI_IMAGE_BILLING wins + * over both. See ProviderRoute.imageBilling. */ + imageBilling?: string; }; function nonEmpty(value: string | undefined): string | undefined { @@ -593,7 +608,7 @@ export function parseRouteEntry(v: unknown): ProviderRoute | undefined { // is the KEY in the providers map (identical to the /bili/ string), // so it is NOT repeated inside the value. if (v && typeof v === "object" && !Array.isArray(v)) { - const obj = v as { models?: Record; proxy?: string; compressProtocol?: string; compress?: CompressSettings; compat?: { roles?: unknown }; passthrough?: boolean }; + const obj = v as { models?: Record; proxy?: string; compressProtocol?: string; compress?: CompressSettings; compat?: { roles?: unknown }; passthrough?: boolean; imageBilling?: unknown }; const route: ProviderRoute = { models: obj.models }; if (typeof obj.proxy === "string") route.proxy = obj.proxy; if (obj.compressProtocol === "marker" || obj.compressProtocol === "tools") route.compressProtocol = obj.compressProtocol; @@ -601,6 +616,8 @@ export function parseRouteEntry(v: unknown): ProviderRoute | undefined { const compatRoles = parseCompatRoles(obj.compat?.roles); if (compatRoles) route.compat = { roles: compatRoles }; if (typeof obj.passthrough === "boolean") route.passthrough = obj.passthrough; + const imageBilling = parseImageBilling(obj.imageBilling); + if (imageBilling) route.imageBilling = imageBilling; return route; } // A bare value (e.g. null) means "this upstream exists, no overrides". @@ -608,6 +625,10 @@ export function parseRouteEntry(v: unknown): ProviderRoute | undefined { return undefined; } +export function parseImageBilling(value: unknown): ImageBillingMode | undefined { + return value === "auto" || value === "pixels" || value === "bytes" ? value : undefined; +} + export function parsePromptCacheRouting(value: string | undefined): PromptCacheRouting { return value === "enabled" || value === "disabled" ? value : "auto"; } diff --git a/src/image-tokens.ts b/src/image-tokens.ts index a8f0361a36..cc89c993f5 100644 --- a/src/image-tokens.ts +++ b/src/image-tokens.ts @@ -5,12 +5,56 @@ // body so callers can add it to the text estimate. Known base64 payloads are // charged ceil(base64Length/4) — the same chars/4 rule defaultCountTokens uses, // matching how byte-counting relays actually bill (#488: 7 screenshots ≈ 1.4M -// tokens); remote URLs we cannot size get a flat conservative cost. Pixel-tile -// upstreams (official Anthropic/OpenAI) will see an overestimate — set -// BILI_IMAGE_TOKEN_CAP to clamp the per-image cost. +// tokens); remote URLs we cannot size get a flat conservative cost. The billing +// mode (see #767 below) decides which of these applies per upstream. +// #767: that overestimate is not just cosmetic — when a session's historical +// lastInputTokens baseline sits above the window, BOTH safe-forward paths close +// at once (#496's forward-once requires a sub-window baseline; #300's +// stale-baseline fit requires payloadEstimate < limit, which base64/4 pushes +// past the window), leaving a 502 loop even though the real image bill is a few +// thousand tokens. The billing mode fixes the estimate at its source: +// bytes — ceil(base64/4), the conservative default for byte-counting relays; +// pixels — parse the container header (PNG/JPEG/WebP/GIF/BMP) for real +// dimensions and charge the OpenAI high-detail tile model +// (85 + 170×tiles, ≤ 2805); unparsable formats fall back to a flat +// PIXEL_IMAGE_FALLBACK_TOKENS. Mode resolution: explicit config wins; +// "auto" classifies known first-party pixel-tile hosts. export const REMOTE_IMAGE_TOKENS = 4096; +/** #767: flat per-image cost in pixels mode when the container has no parsable + * dimensions. Sits well above both published first-party ceilings (OpenAI + * high-detail max 85 + 170×16 = 2805; Anthropic ~1534) while staying far below + * base64/4 for any inline image over ~64KB. */ +export const PIXEL_IMAGE_FALLBACK_TOKENS = 16_384; + +export type ImageBillingMode = "auto" | "pixels" | "bytes"; +export type ResolvedImageBilling = "pixels" | "bytes"; + +/** First-party upstreams that bill images by pixel tiles, not raw bytes. + * Suffix-matched so api.openai.com / *.openai.azure.com / chatgpt.com + * (backend-api/codex) all classify without an explicit override. */ +const PIXEL_TILE_HOSTS = ["api.openai.com", "openai.com", "openai.azure.com", "chatgpt.com", "api.anthropic.com"]; + +/** Resolve a configured mode against the upstream URL. Explicit "pixels"/ + * "bytes" always win; "auto" (or unset) classifies by host — unknown hosts + * stay on the conservative bytes estimate. */ +export function resolveImageBilling(mode: ImageBillingMode | undefined, upstreamUrl: string | undefined): ResolvedImageBilling { + if (mode === "pixels") return "pixels"; + if (mode === "bytes") return "bytes"; + if (!upstreamUrl) return "bytes"; + let host: string; + try { + host = new URL(upstreamUrl).hostname.toLowerCase(); + } catch { + return "bytes"; + } + for (const h of PIXEL_TILE_HOSTS) { + if (host === h || host.endsWith("." + h)) return "pixels"; + } + return "bytes"; +} + function imageTokenCap(): number { const v = Number(process.env.BILI_IMAGE_TOKEN_CAP ?? ""); return Number.isInteger(v) && v > 0 ? v : 0; @@ -21,10 +65,133 @@ function applyCap(cost: number): number { return cap > 0 ? Math.min(cost, cap) : cost; } -function costForUrl(url: string): number { +type Dims = { w: number; h: number }; + +const PNG_SIG = [0x89, 0x50, 0x4e, 0x47]; + +function pngDims(b: Buffer): Dims | undefined { + if (b.length < 24 || !PNG_SIG.every((v, i) => b[i] === v)) return undefined; + const w = b.readUInt32BE(16), h = b.readUInt32BE(20); + return w > 0 && h > 0 ? { w, h } : undefined; +} + +function gifDims(b: Buffer): Dims | undefined { + if (b.length < 10 || b[0] !== 0x47 || b[1] !== 0x49 || b[2] !== 0x46) return undefined; // GIF87a/GIF89a + const w = b.readUInt16LE(6), h = b.readUInt16LE(8); + return w > 0 && h > 0 ? { w, h } : undefined; +} + +function bmpDims(b: Buffer): Dims | undefined { + if (b.length < 26 || b[0] !== 0x42 || b[1] !== 0x4d) return undefined; // "BM" + const w = b.readUInt32LE(18), h = Math.abs(b.readInt32LE(22)); + return w > 0 && h > 0 ? { w, h } : undefined; +} + +function webpDims(b: Buffer): Dims | undefined { + if (b.length < 30 || b.toString("ascii", 0, 4) !== "RIFF" || b.toString("ascii", 8, 12) !== "WEBP") return undefined; + const fourcc = b.toString("ascii", 12, 16); + // RIFF(12) + FourCC(4) + ChunkSize(4) → chunk data starts at offset 20. + if (fourcc === "VP8 ") { + // RFC 6386 §9.1: frame tag(3) + sync code 9D 01 2A, then 14-bit dims. + if (b[23] !== 0x9d || b[24] !== 0x01 || b[25] !== 0x2a) return undefined; + const w = b.readUInt16LE(26) & 0x3fff, h = b.readUInt16LE(28) & 0x3fff; + return w > 0 && h > 0 ? { w, h } : undefined; + } + if (fourcc === "VP8L") { + if (b[20] !== 0x2f) return undefined; // lossless signature byte + // 32-bit little-endian pack after the signature: width-1 (14 bits) | + // height-1 (14 bits) | version (4 bits) — width and height share byte 22. + const w = ((b[22] & 0x3f) << 8 | b[21]) + 1, h = (((b[24] & 0x0f) << 10) | (b[23] << 2) | ((b[22] & 0xc0) >> 6)) + 1; + return w > 0 && h > 0 ? { w, h } : undefined; + } + if (fourcc === "VP8X") { + const w = (b[24] | (b[25] << 8) | (b[26] << 16)) + 1, h = (b[27] | (b[28] << 8) | (b[29] << 16)) + 1; + return w > 0 && h > 0 ? { w, h } : undefined; + } + return undefined; +} + +function jpegDims(b: Buffer): Dims | undefined { + if (b.length < 4 || b[0] !== 0xff || b[1] !== 0xd8) return undefined; + let off = 2; + while (off + 4 <= b.length) { + if (b[off] !== 0xff) { off += 1; continue; } + const marker = b[off + 1]; + if (marker === 0xd8 || marker === 0x01 || (marker >= 0xd0 && marker <= 0xd9)) { off += 2; continue; } + if (marker === 0xda) break; // SOS with no SOF before it — malformed + const segLen = b.readUInt16BE(off + 2); + if (segLen < 2) break; + // SOFn (C0-C3, C5-C7, C9-CB, CD-CF): precision(1) height(2) width(2)… + if (marker >= 0xc0 && marker <= 0xcf && marker !== 0xc4 && marker !== 0xc8 && marker !== 0xcc) { + if (off + 9 > b.length) break; + const h = b.readUInt16BE(off + 5), w = b.readUInt16BE(off + 7); + if (w > 0 && h > 0) return { w, h }; + break; + } + off += 2 + segLen; + } + return undefined; +} + +// Decode budgets: everything except JPEG keeps its dimensions within the first +// 32 bytes; JPEG SOF can sit behind large EXIF APP segments, so scan deeper — +// capping the decode at ~260KB keeps the worst case sub-millisecond. +const HEADER_SCAN_CHARS = 64; +const JPEG_SCAN_CHARS = 350_000; + +/** Parse image dimensions from a base64 payload WITHOUT decoding the whole + * image (only the leading decode budget). Returns undefined for unknown or + * truncated containers — callers fall back to the flat pixels-mode cost. */ +export function decodeImageDims(b64: string): Dims | undefined { + const head = Buffer.from(b64.slice(0, HEADER_SCAN_CHARS), "base64"); + for (const parse of [pngDims, gifDims, bmpDims, webpDims]) { + const d = parse(head); + if (d) return d; + } + if (head.length >= 2 && head[0] === 0xff && head[1] === 0xd8) { + return jpegDims(Buffer.from(b64.slice(0, JPEG_SCAN_CHARS), "base64")); + } + return undefined; +} + +const TILE_EDGE = 512; +const TILE_SHORT_SIDE = 768; +const TILE_MAX_EDGE = 2048; + +/** OpenAI high-detail tile model (published formula): short side scaled UP to + * 768, long side capped at 2048, then 512px tiles at 170 tokens each plus an + * 85-token base. Bounds: [~765, 2805]. Anthropic's own model (area ≤ 1.568M + * px², 170/tile + 4) tops out near 1534, so this overestimates both + * first-party billings for typical screenshots — the safe direction for gate + * decisions (an undercount would forward a genuinely over-window payload). */ +export function pixelTileEstimate(w: number, h: number): number { + let sw = w, sh = h; + const shortSide = Math.min(sw, sh); + if (shortSide > 0 && shortSide < TILE_SHORT_SIDE) { + const s = TILE_SHORT_SIDE / shortSide; + sw *= s; sh *= s; + } + const longSide = Math.max(sw, sh); + if (longSide > TILE_MAX_EDGE) { + const s = TILE_MAX_EDGE / longSide; + sw *= s; sh *= s; + } + const tiles = Math.ceil(sw / TILE_EDGE) * Math.ceil(sh / TILE_EDGE); + return 85 + 170 * tiles; +} + +function base64ImageCost(b64: string, billing: ResolvedImageBilling): number { + if (billing === "pixels") { + const dims = decodeImageDims(b64); + return applyCap(dims ? pixelTileEstimate(dims.w, dims.h) : PIXEL_IMAGE_FALLBACK_TOKENS); + } + return applyCap(Math.ceil(b64.length / 4)); +} + +function costForUrl(url: string, billing: ResolvedImageBilling): number { if (url.startsWith("data:")) { const i = url.indexOf("base64,"); - if (i >= 0) return applyCap(Math.ceil((url.length - i - "base64,".length) / 4)); + if (i >= 0) return base64ImageCost(url.slice(i + "base64,".length), billing); } return applyCap(REMOTE_IMAGE_TOKENS); } @@ -39,7 +206,7 @@ function urlOf(v: unknown): string | undefined { return undefined; } -export function imageTokensInParsedBody(protocol: "anthropic" | "openai" | "responses", body: unknown): number { +export function imageTokensInParsedBody(protocol: "anthropic" | "openai" | "responses", body: unknown, billing: ResolvedImageBilling = "bytes"): number { if (!isObj(body)) return 0; let total = 0; if (protocol === "responses") { @@ -50,7 +217,7 @@ export function imageTokensInParsedBody(protocol: "anthropic" | "openai" | "resp for (const part of item.content) { if (!isObj(part) || part.type !== "input_image") continue; const url = urlOf(part.image_url); - if (url) total += costForUrl(url); + if (url) total += costForUrl(url, billing); } } return total; @@ -64,12 +231,12 @@ export function imageTokensInParsedBody(protocol: "anthropic" | "openai" | "resp if (protocol === "openai") { if (part.type !== "image_url") continue; const url = urlOf(part.image_url); - if (url) total += costForUrl(url); + if (url) total += costForUrl(url, billing); } else { if (part.type !== "image") continue; const src = part.source; - if (isObj(src) && src.type === "base64" && typeof src.data === "string") total += applyCap(Math.ceil(src.data.length / 4)); - else if (isObj(src) && src.type === "url" && typeof src.url === "string") total += costForUrl(src.url); + if (isObj(src) && src.type === "base64" && typeof src.data === "string") total += base64ImageCost(src.data, billing); + else if (isObj(src) && src.type === "url" && typeof src.url === "string") total += costForUrl(src.url, billing); } } } @@ -79,7 +246,7 @@ export function imageTokensInParsedBody(protocol: "anthropic" | "openai" | "resp // Cheap gate: most bodies carry no images — skip the JSON parse entirely then. // prepared.body is bili's own compact JSON.stringify, but client raw buffers // may carry spaces, so probe both forms. -export function imageTokensInRawBody(protocol: "anthropic" | "openai" | "responses", raw: string | Buffer): number { +export function imageTokensInRawBody(protocol: "anthropic" | "openai" | "responses", raw: string | Buffer, billing: ResolvedImageBilling = "bytes"): number { const s = typeof raw === "string" ? raw : raw.toString("utf8"); const probe = protocol === "responses" ? s.includes("input_image") @@ -87,7 +254,7 @@ export function imageTokensInRawBody(protocol: "anthropic" | "openai" | "respons : s.includes('"type":"image"') || s.includes('"type": "image"'); if (!probe) return 0; try { - return imageTokensInParsedBody(protocol, JSON.parse(s)); + return imageTokensInParsedBody(protocol, JSON.parse(s), billing); } catch { return 0; } diff --git a/src/server.ts b/src/server.ts index 9f723ddc80..3f64a711a3 100644 --- a/src/server.ts +++ b/src/server.ts @@ -49,7 +49,7 @@ import { applyAbsorbView, absorbEnabled, absorbToolName, storeEffectiveAbsorb } import { rewriteJsonResponse, type RewriteCtx } from "./stream.js"; import { applyRanges } from "./stream.js"; import { preflightCompress, estimateCoreMessages, estimateRawBodyTokens, estimateCoreMessagesUpper, type PreflightResult } from "./preflight.js"; -import { imageTokensInRawBody, imageTokensInParsedBody } from "./image-tokens.js"; +import { imageTokensInRawBody, imageTokensInParsedBody, resolveImageBilling, type ResolvedImageBilling } from "./image-tokens.js"; import { renderUI, handleConfigGet, handleConfigPut } from "./web/index.js"; import { reapOrphanBlocks } from "./orphan-gc.js"; import { getStore } from "./persist.js"; @@ -718,16 +718,28 @@ export function sideRequestGuard( protocol: WireProtocol, modelContextLimit: number, learnedLimit: number | undefined, + imageBilling: ResolvedImageBilling = "bytes", ): { blocked: boolean; estimate: number; limit: number } { let limit = modelContextLimit; if (typeof learnedLimit === "number" && learnedLimit > 0 && learnedLimit < limit) limit = learnedLimit; const field = outputBudgetField(parsed); const maxOut = field ? ((parsed as Record)[field] as number) : 0; if (limit > 0 && shouldReserveOutputHeadroom(protocol)) limit = reserveOutputHeadroom(limit, maxOut); - const estimate = estimateRawBodyTokens(parsed) + imageTokensInParsedBody(protocol, parsed); + const estimate = estimateRawBodyTokens(parsed) + imageTokensInParsedBody(protocol, parsed, imageBilling); return { blocked: limit > 0 && estimate >= limit * SIDE_REQUEST_GUARD_TOLERANCE, estimate, limit }; } +// #767: per-request image billing mode — env BILI_IMAGE_BILLING (live, like +// BILI_IMAGE_TOKEN_CAP) wins over the per-provider route entry, which wins over +// the global config level; "auto"/unset classifies known first-party pixel-tile +// hosts by upstream URL. Every payload-size decision below consults this so one +// over-estimate cannot block all of them at once. +function imageBillingFor(opts: ProxyOptions, upstreamUrl: string | undefined): ResolvedImageBilling { + const env = process.env.BILI_IMAGE_BILLING; + const configured = env === "pixels" || env === "bytes" ? env : findRoute(opts.routes, upstreamUrl)?.imageBilling ?? opts.imageBilling ?? "auto"; + return resolveImageBilling(configured, upstreamUrl); +} + function isTrustedAdminOrigin(origin: string | undefined, host: string | undefined, trustedHosts: Set): boolean { // Host must be one of OUR listen identities regardless of whether an // Origin header is present. A same-origin browser GET/fetch (the DNS @@ -1449,7 +1461,7 @@ async function handle( // then scalar) — the learner now writes the confirmed channel, so the // legacy direct map read would miss windows learned from real 400s. const learnedLimit = resolveLearnedLimit(session, reqModel); - const guard = sideRequestGuard(parsed, protocol, reqConfig.modelContextLimit, learnedLimit); + const guard = sideRequestGuard(parsed, protocol, reqConfig.modelContextLimit, learnedLimit, imageBillingFor(opts, route?.rewrittenUrl ?? upstreamOrigin)); if (guard.blocked) { log("warn", `[${session.id}] side request (~${guard.estimate} tokens) ≥ effective window ${guard.limit} (model=${reqModel ?? "?"}) — NOT forwarded: guaranteed upstream 400 (side requests bypass preflight by design, #388)`); if (!res.headersSent && !res.writableEnded && !res.destroyed) { @@ -1604,14 +1616,14 @@ async function handle( : protocol === "anthropic" ? prepareAnthropic(work as AnthropicRequestBody, req, opts, core, reqConfig, reqPrompts, reqSurface, log, session, pluginMode, upstreamOrigin, reasoningCfg) : protocol === "openai" - ? prepareOpenai(work as OpenAIRequestBody, req, opts, core, reqConfig, reqPrompts, reqSurface, log, session, pluginMode, upstreamOrigin, nativeWindow, reasoningCfg) + ? prepareOpenai(work as OpenAIRequestBody, req, opts, core, reqConfig, reqPrompts, reqSurface, log, session, pluginMode, upstreamOrigin, nativeWindow, reasoningCfg, route?.rewrittenUrl) : responsesCompact // #618 review nit: when no bili compaction item is present, // prepareResponsesCompact falls back to the raw bodyBuffer — forward // the re-serialized post-strip work instead so dropped images don't // ride along. Unchanged bodies keep the original buffer byte-identical. ? prepareResponsesCompact(stripped.removed > 0 ? Buffer.from(JSON.stringify(work)) : bodyBuffer, work as ResponsesRequestBody, session, req, core, reqConfig, log) - : prepareResponses(work as ResponsesRequestBody, req, opts, core, reqConfig, reqPrompts, reqSurface, log, session, responsesIdentity!, pluginMode, upstreamOrigin, nativeWindow, reasoningCfg); + : prepareResponses(work as ResponsesRequestBody, req, opts, core, reqConfig, reqPrompts, reqSurface, log, session, responsesIdentity!, pluginMode, upstreamOrigin, nativeWindow, reasoningCfg, route?.rewrittenUrl); }; // #332: codex's native remote-compaction request (trigger form) // is dispatched BEFORE prepare/preflight. When it is not @@ -2203,6 +2215,7 @@ function prepareOpenai( upstreamOrigin: string, nativeWindow: number, reasoning: CompressReasoningConfig | undefined, + billingUpstream?: string, ): Prepared { const sessionId = session.id; const stream = parsed.stream === true; @@ -2316,7 +2329,7 @@ function prepareOpenai( const rebuilt: OpenAIRequestBody = { ...parsed, messages: rebuiltMessages, tools: toolsOut as OpenAITool[] | undefined }; warnReasoningPairs(rebuiltMessages, log, sessionId); - clampOutgoingOutput(rebuilt as Record, typeof (parsed as Record).max_completion_tokens === "number" ? "max_completion_tokens" : "max_tokens", { systemText: openaiSystemText, tools: toolsOut, processedMessages, lastInputTokens: session.stats.lastInputTokens, nativeWindow, imageTokens: imageTokensInParsedBody("openai", rebuilt) }, sessionId, log); + clampOutgoingOutput(rebuilt as Record, typeof (parsed as Record).max_completion_tokens === "number" ? "max_completion_tokens" : "max_tokens", { systemText: openaiSystemText, tools: toolsOut, processedMessages, lastInputTokens: session.stats.lastInputTokens, nativeWindow, imageTokens: imageTokensInParsedBody("openai", rebuilt, imageBillingFor(opts, billingUpstream ?? upstreamOrigin)) }, sessionId, log); // prompt_cache_retention is an OpenAI-host-only cache directive; the dsh // launcher forces PI_CACHE_RETENTION=long (for the session-id // prompt_cache_key) which makes the client also emit it. Third-party @@ -2359,6 +2372,7 @@ function prepareResponses( upstreamOrigin: string, nativeWindow: number, reasoning: CompressReasoningConfig | undefined, + billingUpstream?: string, ): Prepared { const sessionId = session.id; const stream = parsed.stream === true; @@ -2551,7 +2565,7 @@ function prepareResponses( const rebuilt: ResponsesRequestBody = { ...parsed, input: rebuiltInput, tools: toolsOut }; warnResponsesReasoningPairs(Array.isArray(rebuiltInput) ? rebuiltInput : [], log, sessionId); if (!isCompactionTrigger) { - clampOutgoingOutput(rebuilt as Record, "max_output_tokens", { systemText: (responsesProjection?.systemParts ?? []).join("\n"), tools: toolsOut, processedMessages, lastInputTokens: session.stats.lastInputTokens, nativeWindow, imageTokens: imageTokensInParsedBody("responses", rebuilt) }, sessionId, log); + clampOutgoingOutput(rebuilt as Record, "max_output_tokens", { systemText: (responsesProjection?.systemParts ?? []).join("\n"), tools: toolsOut, processedMessages, lastInputTokens: session.stats.lastInputTokens, nativeWindow, imageTokens: imageTokensInParsedBody("responses", rebuilt, imageBillingFor(opts, billingUpstream ?? upstreamOrigin)) }, sessionId, log); } // Route with the upstream THIS request goes to — session.meta.upstreamOrigin // is first-wins and would keep injecting pck toward a relay we switched @@ -3075,7 +3089,10 @@ async function preflightCompressIfNeeded( // trigger on the real post-fold payload too. // #488: images are forwarded verbatim but invisible to the kernel's text model — // add their cost to every size decision here (trigger, fit gates, self-heal). - const imageTokens = imageTokensInRawBody(prepared.protocol, prepared.body); + // #767: bill them by the resolved mode (same upstream-URL fallback as buildForwardTarget). + const preflightReqUrl = req.url ?? ""; + const billingUpstream = route?.rewrittenUrl ?? (/^https?:\/\//i.test(preflightReqUrl) ? preflightReqUrl : opts.upstream); + const imageTokens = imageTokensInRawBody(prepared.protocol, prepared.body, imageBillingFor(opts, billingUpstream)); const textEstimate = estimateCoreMessages(prepared.processedMessages); // #470: system + tool definitions ride the wire too but are invisible to // estimateCoreMessages — without them the trigger fires late (text alone @@ -3639,7 +3656,7 @@ async function forward( parsedBody = JSON.parse(rawBody) as Record; reqModel = typeof parsedBody.model === "string" ? parsedBody.model : undefined; // #488: the rejected payload's size must include its images (they were forwarded verbatim). - rejectedImageTokens = imageTokensInParsedBody(prepared.protocol, parsedBody); + rejectedImageTokens = imageTokensInParsedBody(prepared.protocol, parsedBody, imageBillingFor(opts, upstreamUrl)); } catch { reqModel = undefined; // non-JSON body — fall back to the legacy scalar } @@ -4252,6 +4269,7 @@ function handleConfigReload(opts: ProxyOptions, res: http.ServerResponse, log: ( Object.assign(opts.routes, fresh); opts.compress = loadOptions().compress; opts.compat = loadOptions().compat; + opts.imageBilling = loadOptions().imageBilling; // Release cached ProxyAgents so agents for proxy URLs that were // removed/changed don't leak for the process lifetime. The next request // re-creates the needed agent lazily via proxyDispatcher(). diff --git a/tests/issue767-pixel-billing.test.ts b/tests/issue767-pixel-billing.test.ts new file mode 100644 index 0000000000..e0a518a21a --- /dev/null +++ b/tests/issue767-pixel-billing.test.ts @@ -0,0 +1,453 @@ +import assert from "node:assert/strict"; +import http from "node:http"; +import { once } from "node:events"; +import test from "node:test"; + +process.env.NODE_ENV = "test"; + +import { defaultConfig } from "acp-kernel"; +import { startServer, type ProxyOptions } from "../src/server.ts"; +import { SessionStore, _setStoreForTest } from "../src/persist.ts"; +import { _setForTest as setRegistryForTest } from "../src/registry.ts"; +import { listSessions } from "../src/session.ts"; +import { + PIXEL_IMAGE_FALLBACK_TOKENS, + REMOTE_IMAGE_TOKENS, + decodeImageDims, + imageTokensInParsedBody, + imageTokensInRawBody, + pixelTileEstimate, + resolveImageBilling, +} from "../src/image-tokens.ts"; + +// Issue #767 (follow-up to #496/#500): on pixel-tile billing upstreams (official +// Codex/OpenAI), the default base64/4 image cost overestimates real tile billing +// by orders of magnitude. When a session's historical lastInputTokens baseline +// already sits above the window, that overestimate closes BOTH safe-forward paths +// at once (#496 forward-once requires a sub-window baseline; #300 stale-baseline +// fit requires payloadEstimate < limit) → a persistent 502 loop even though the +// images bill only a few thousand tokens. Fix: per-provider `imageBilling` +// ("pixels" | "bytes", env BILI_IMAGE_BILLING overrides both) charges by a +// dimension-based tile model instead of raw bytes; bytes stays the conservative +// default so byte-counting relays keep #488/#496 protection intact. + +const WINDOW = 272_000; // gpt-6-astra effective limit from the issue +const STALE_BASELINE = 696_789; // the reported stuck lastInputTokens + +function pngB64(w: number, h: number, padChars = 0): string { + const b = Buffer.alloc(24); + b.set([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a], 0); + b.writeUInt32BE(13, 8); + b.write("IHDR", 12, "ascii"); + b.writeUInt32BE(w, 16); + b.writeUInt32BE(h, 20); + return b.toString("base64") + "A".repeat(padChars); +} + +function gifB64(w: number, h: number): string { + const b = Buffer.alloc(13); + b.write("GIF89a", 0, "ascii"); + b.writeUInt16LE(w, 6); + b.writeUInt16LE(h, 8); + return b.toString("base64"); +} + +function bmpB64(w: number, h: number): string { + const b = Buffer.alloc(26); + b.write("BM", 0, "ascii"); + b.writeUInt32LE(26, 2); + b.writeUInt32LE(26, 10); + b.writeUInt32LE(40, 14); + b.writeInt32LE(w, 18); + b.writeInt32LE(h, 22); + return b.toString("base64"); +} + +function webpVp8B64(w: number, h: number): string { + const b = Buffer.alloc(32); + b.write("RIFF", 0, "ascii"); + b.writeUInt32LE(32 - 8, 4); + b.write("WEBP", 8, "ascii"); + b.write("VP8 ", 12, "ascii"); + b.writeUInt32LE(12, 16); + b[20] = 0x30; + b[21] = 0x00; + b[22] = 0x00; + b[23] = 0x9d; + b[24] = 0x01; + b[25] = 0x2a; + b.writeUInt16LE(w, 26); + b.writeUInt16LE(h, 28); + return b.toString("base64"); +} + +function webpVp8lB64(w: number, h: number): string { + const wm = w - 1, hm = h - 1; + const b = Buffer.alloc(32); + b.write("RIFF", 0, "ascii"); + b.writeUInt32LE(32 - 8, 4); + b.write("WEBP", 8, "ascii"); + b.write("VP8L", 12, "ascii"); + b.writeUInt32LE(12, 16); + b[20] = 0x2f; + b[21] = wm & 0xff; + b[22] = ((wm >> 8) & 0x3f) | ((hm & 0x3) << 6); + b[23] = (hm >> 2) & 0xff; + b[24] = (hm >> 10) & 0x0f; + return b.toString("base64"); +} + +function webpVp8xB64(w: number, h: number): string { + const b = Buffer.alloc(32); + b.write("RIFF", 0, "ascii"); + b.writeUInt32LE(32 - 8, 4); + b.write("WEBP", 8, "ascii"); + b.write("VP8X", 12, "ascii"); + b.writeUInt32LE(10, 16); + b[20] = 0x00; + b[24] = (w - 1) & 0xff; + b[25] = ((w - 1) >> 8) & 0xff; + b[26] = ((w - 1) >> 16) & 0xff; + b[27] = (h - 1) & 0xff; + b[28] = ((h - 1) >> 8) & 0xff; + b[29] = ((h - 1) >> 16) & 0xff; + return b.toString("base64"); +} + +function jpegB64(w: number, h: number): string { + const sof = Buffer.from([0xff, 0xc0, 0x00, 0x11, 0x08]); + const dims = Buffer.alloc(4); + dims.writeUInt16BE(h, 0); + dims.writeUInt16BE(w, 2); + const b = Buffer.concat([ + Buffer.from([0xff, 0xd8]), + Buffer.from([0xff, 0xe0, 0x00, 0x10]), + Buffer.alloc(14), + Buffer.from([0xff, 0xe1, 0x01, 0x00]), + Buffer.alloc(254), + sof, + dims, + Buffer.alloc(4, 0x00), + ]); + return b.toString("base64"); +} + +test("#767 unit: decodeImageDims parses container headers without full decode", () => { + assert.deepEqual(decodeImageDims(pngB64(2048, 1536)), { w: 2048, h: 1536 }); + assert.deepEqual(decodeImageDims(gifB64(320, 240)), { w: 320, h: 240 }); + assert.deepEqual(decodeImageDims(bmpB64(640, 480)), { w: 640, h: 480 }); + assert.deepEqual(decodeImageDims(webpVp8B64(1280, 720)), { w: 1280, h: 720 }); + assert.deepEqual(decodeImageDims(webpVp8lB64(1000, 800)), { w: 1000, h: 800 }); + assert.deepEqual(decodeImageDims(webpVp8xB64(300, 200)), { w: 300, h: 200 }); + assert.deepEqual(decodeImageDims(jpegB64(512, 384)), { w: 512, h: 384 }, "JPEG dims found behind APP0+EXIF padding"); + assert.equal(decodeImageDims("A".repeat(64)), undefined, "unparsable garbage falls back to flat cost"); + assert.equal(decodeImageDims(Buffer.from([0x89, 0x50, 0x4e, 0x47]).toString("base64")), undefined, "truncated header → fallback"); +}); + +test("#767 unit: pixelTileEstimate matches the OpenAI high-detail tile model", () => { + assert.equal(pixelTileEstimate(100, 100), 765, "small square scales up to 768×768 = 4 tiles"); + assert.equal(pixelTileEstimate(2048, 2048), 2805, "max-size square = 16 tiles (the ceiling)"); + assert.equal(pixelTileEstimate(3584, 1024), 1445, "long side capped at 2048 → 4×2 tiles"); + assert.equal(pixelTileEstimate(1000, 200), 765, "double scale: up short side, then cap long side"); +}); + +test("#767 unit: resolveImageBilling — explicit wins, auto classifies first-party hosts", () => { + assert.equal(resolveImageBilling("pixels", undefined), "pixels"); + assert.equal(resolveImageBilling("bytes", "https://api.openai.com/v1"), "bytes"); + assert.equal(resolveImageBilling(undefined, "https://api.openai.com/v1"), "pixels"); + assert.equal(resolveImageBilling("auto", "https://chatgpt.com/backend-api/codex/responses"), "pixels"); + assert.equal(resolveImageBilling("auto", "mitm://chatgpt.com/backend-api/codex/responses"), "pixels"); + assert.equal(resolveImageBilling("auto", "https://api.anthropic.com/v1/messages"), "pixels"); + assert.equal(resolveImageBilling("auto", "https://westus2.openai.azure.com/openai/v1"), "pixels"); + assert.equal(resolveImageBilling("auto", "https://open.bigmodel.cn/api/paas/v4"), "bytes"); + assert.equal(resolveImageBilling("auto", "https://evil-openai.com/v1"), "bytes", "suffix match must not hit lookalike hosts"); + assert.equal(resolveImageBilling("auto", undefined), "bytes"); + assert.equal(resolveImageBilling("auto", "not-a-url"), "bytes"); +}); + +test("#767 unit: parsed-body costs — pixels vs bytes, fallback, remote URLs, cap", () => { + const big = pngB64(2048, 1536, 600_000); + const body = { input: [{ type: "message", role: "user", content: [{ type: "input_text", text: "hi" }, { type: "input_image", image_url: `data:image/png;base64,${big}` }] }] }; + const bytesCost = Math.ceil(big.length / 4); + assert.ok(bytesCost > WINDOW / 2, "fixture really overflows the window under byte billing"); + assert.equal(imageTokensInParsedBody("responses", body, "bytes"), bytesCost); + // 2048×1536 → tiles ceil(2048/512)*ceil(1536/512) = 4*3 = 12 → 85 + 170*12 + assert.equal(imageTokensInParsedBody("responses", body, "pixels"), 85 + 170 * 12); + + const remote = { input: [{ type: "message", role: "user", content: [{ type: "input_image", image_url: "https://example.com/screenshot.png" }] }] }; + assert.equal(imageTokensInParsedBody("responses", remote, "pixels"), REMOTE_IMAGE_TOKENS); + assert.equal(imageTokensInParsedBody("responses", remote, "bytes"), REMOTE_IMAGE_TOKENS); + + const unknownFmt = { input: [{ type: "message", role: "user", content: [{ type: "input_image", image_url: `data:image/png;base64,${"A".repeat(4000)}` }] }] }; + assert.equal(imageTokensInParsedBody("responses", unknownFmt, "pixels"), PIXEL_IMAGE_FALLBACK_TOKENS); + assert.equal(imageTokensInParsedBody("responses", unknownFmt, "bytes"), 1000); + + process.env.BILI_IMAGE_TOKEN_CAP = "500"; + try { + assert.equal(imageTokensInParsedBody("responses", body, "pixels"), 500, "cap clamps pixels mode too"); + assert.equal(imageTokensInParsedBody("responses", body, "bytes"), 500); + } finally { + delete process.env.BILI_IMAGE_TOKEN_CAP; + } +}); + +test("#767 unit: raw-body probe honors the billing mode", () => { + const big = pngB64(3584, 1024, 600_000); + const raw = JSON.stringify({ input: [{ type: "message", role: "user", content: [{ type: "input_image", image_url: `data:image/png;base64,${big}` }] }] }); + assert.equal(imageTokensInRawBody("responses", raw, "pixels"), 1445); + assert.ok(imageTokensInRawBody("responses", raw, "bytes") > WINDOW / 2); + assert.equal(imageTokensInRawBody("responses", JSON.stringify({ input: [] }), "pixels"), 0); +}); + +type MockStats = { streamingForwards: number; imagesSeen: number; summaryCalls: number }; + +function startMockUpstream(onStream?: (raw: string) => void): Promise<{ server: http.Server; port: number; stats: MockStats }> { + const stats: MockStats = { streamingForwards: 0, imagesSeen: 0, summaryCalls: 0 }; + const server = http.createServer((req, res) => { + const chunks: Buffer[] = []; + req.on("data", (c: Buffer) => chunks.push(c)); + req.on("end", () => { + const raw = Buffer.concat(chunks).toString("utf8"); + let parsed: { stream?: boolean } = {}; + try { parsed = JSON.parse(raw); } catch { parsed = {}; } + if (parsed.stream === false) { + stats.summaryCalls += 1; + res.writeHead(200, { "content-type": "application/json" }); + res.end(JSON.stringify({ output_text: "PREFLIGHT SUMMARY: folded segment." })); + return; + } + stats.streamingForwards += 1; + stats.imagesSeen += (raw.match(/"input_image"/g) ?? []).length; + onStream?.(raw); + res.writeHead(200, { "content-type": "text/event-stream", "cache-control": "no-cache" }); + res.write(`event: response.completed\ndata: ${JSON.stringify({ response: { id: "resp_done", status: "completed", output: [], usage: { input_tokens: 6617, output_tokens: 5, total_tokens: 6622 } } })}\n\n`); + res.end(); + }); + }); + server.listen(0, "127.0.0.1"); + return new Promise((resolve) => { + server.on("listening", () => resolve({ server, port: (server.address() as { port: number }).port, stats })); + }); +} + +async function startProxy(upstreamPort: number, routeExtra: Record = {}, routeKey?: string): Promise<{ proxy: http.Server; port: number }> { + _setStoreForTest(new SessionStore({ enabled: false })); + setRegistryForTest({}); + const proxy = await startServer({ + port: 0, + host: "127.0.0.1", + upstream: "http://127.0.0.1", + routes: { [routeKey ?? `http://127.0.0.1:${upstreamPort}`]: { models: { "gpt-astra": { context: WINDOW } }, ...routeExtra } }, + modelContextLimit: WINDOW, + kernelConfig: defaultConfig(WINDOW), + compress: { injectTool: true, injectNudge: true }, + promptCache: { routing: "auto" }, + sessionHeader: "x-acp-session", + log: false, + debug: false, + passthrough: false, + autoUpdate: false, + mitm: { enabled: false, domains: [] }, + } as ProxyOptions); + await once(proxy, "listening"); + return { proxy, port: (proxy.address() as { port: number }).port }; +} + +function imageTurn(sessionId: string): string { + const img = (w: number, h: number) => ({ type: "input_image", image_url: `data:image/png;base64,${pngB64(w, h, 600_000)}` }); + return JSON.stringify({ + model: "gpt-astra", + stream: true, + store: false, + session_id: sessionId, + instructions: "You are the test coding agent.", + input: [{ type: "message", role: "user", content: [{ type: "input_text", text: "two screenshots attached" }, img(3584, 1024), img(3584, 1024)] }], + max_output_tokens: 1024, + }); +} + +test("e2e #767: pixels billing recovers a stale-baseline session the bytes estimate bricks", async () => { + const { server: upstream, port: upstreamPort, stats } = await startMockUpstream(); + const { proxy, port: proxyPort } = await startProxy(upstreamPort, { imageBilling: "pixels" }); + try { + const url = `http://127.0.0.1:${proxyPort}/bili/http://127.0.0.1:${upstreamPort}/v1/responses`; + const headers = { "content-type": "application/json" }; + + const r1 = await fetch(url, { + method: "POST", + headers, + body: JSON.stringify({ model: "gpt-astra", stream: true, store: false, session_id: "img-pix-sess", instructions: "You are the test coding agent.", input: [{ type: "message", role: "user", content: "hello there" }], max_output_tokens: 1024 }), + }); + assert.equal(r1.status, 200); + await r1.text(); + + const s = listSessions().find((x) => x.meta.label === "img-pix-sess"); + assert.ok(s, "session established"); + s!.stats.lastInputTokens = STALE_BASELINE; + + const r2 = await fetch(url, { method: "POST", headers, body: imageTurn("img-pix-sess") }); + assert.equal(r2.status, 200, "stale baseline no longer bricks pixel-billing payloads"); + await r2.text(); + assert.equal(stats.streamingForwards, 2, "both turns were forwarded"); + assert.equal(stats.imagesSeen, 2, "both screenshots reached the upstream verbatim"); + + const after = listSessions().find((x) => x.meta.label === "img-pix-sess")!; + assert.ok(after.stats.lastInputTokens > 0 && after.stats.lastInputTokens < WINDOW, `baseline recovered to real usage (got ${after.stats.lastInputTokens})`); + + const r3 = await fetch(url, { method: "POST", headers, body: imageTurn("img-pix-sess") }); + assert.equal(r3.status, 200, "steady state keeps working after recovery"); + await r3.text(); + assert.equal(stats.streamingForwards, 3); + assert.equal(stats.imagesSeen, 4); + } finally { + proxy.close(); + await once(proxy, "close"); + upstream.close(); + await once(upstream, "close"); + } +}); + +test("e2e #767 control: default (auto→bytes on unknown host) still fails fast — relay protection intact", async () => { + const { server: upstream, port: upstreamPort, stats } = await startMockUpstream(); + const { proxy, port: proxyPort } = await startProxy(upstreamPort); + try { + const url = `http://127.0.0.1:${proxyPort}/bili/http://127.0.0.1:${upstreamPort}/v1/responses`; + const headers = { "content-type": "application/json" }; + + const r1 = await fetch(url, { + method: "POST", + headers, + body: JSON.stringify({ model: "gpt-astra", stream: true, store: false, session_id: "img-byt-sess", instructions: "You are the test coding agent.", input: [{ type: "message", role: "user", content: "hello there" }], max_output_tokens: 1024 }), + }); + assert.equal(r1.status, 200); + await r1.text(); + + const s = listSessions().find((x) => x.meta.label === "img-byt-sess"); + assert.ok(s); + s!.stats.lastInputTokens = STALE_BASELINE; + + const r2 = await fetch(url, { method: "POST", headers, body: imageTurn("img-byt-sess") }); + assert.equal(r2.status, 502, "byte-billed over-window images stay blocked locally"); + const err2 = JSON.parse(await r2.text()) as { error?: { code?: string; message?: string } }; + assert.equal(err2.error?.code, "preflight_compress_failed"); + assert.ok(err2.error?.message?.includes("Images alone account for"), "fail-fast carries the image remedy wording"); + assert.equal(stats.streamingForwards, 1, "only the text-only turn was ever forwarded"); + assert.equal(stats.imagesSeen, 0, "over-window images never reached the upstream"); + } finally { + proxy.close(); + await once(proxy, "close"); + upstream.close(); + await once(upstream, "close"); + } +}); + +test("e2e #767: learned-limit-only variant also closes forward-once (bytes mode)", async () => { + const { server: upstream, port: upstreamPort, stats } = await startMockUpstream(); + const { proxy, port: proxyPort } = await startProxy(upstreamPort); + try { + const url = `http://127.0.0.1:${proxyPort}/bili/http://127.0.0.1:${upstreamPort}/v1/responses`; + const headers = { "content-type": "application/json" }; + + const r1 = await fetch(url, { + method: "POST", + headers, + body: JSON.stringify({ model: "gpt-astra", stream: true, store: false, session_id: "img-lrn-sess", instructions: "You are the test coding agent.", input: [{ type: "message", role: "user", content: "hello there" }], max_output_tokens: 1024 }), + }); + assert.equal(r1.status, 200); + await r1.text(); + + const s = listSessions().find((x) => x.meta.label === "img-lrn-sess"); + assert.ok(s); + assert.ok(s!.stats.lastInputTokens < WINDOW, "baseline is fresh here"); + (s!.metadata as Record).confirmedContextLimits = { "gpt-astra": WINDOW }; + + const r2 = await fetch(url, { method: "POST", headers, body: imageTurn("img-lrn-sess") }); + assert.equal(r2.status, 502, "overflow evidence (learned limit) closes #496 forward-once even with a fresh baseline"); + const err2 = JSON.parse(await r2.text()) as { error?: { code?: string } }; + assert.equal(err2.error?.code, "preflight_compress_failed"); + assert.equal(stats.streamingForwards, 1, "image-bearing turn was never forwarded"); + assert.equal(stats.imagesSeen, 0); + } finally { + proxy.close(); + await once(proxy, "close"); + upstream.close(); + await once(upstream, "close"); + } +}); + +test("e2e #767: env BILI_IMAGE_BILLING beats per-provider route config", async () => { + const { server: upstream, port: upstreamPort, stats } = await startMockUpstream(); + const { proxy, port: proxyPort } = await startProxy(upstreamPort, { imageBilling: "pixels" }); + try { + const url = `http://127.0.0.1:${proxyPort}/bili/http://127.0.0.1:${upstreamPort}/v1/responses`; + const headers = { "content-type": "application/json" }; + + const r1 = await fetch(url, { + method: "POST", + headers, + body: JSON.stringify({ model: "gpt-astra", stream: true, store: false, session_id: "img-env-sess", instructions: "You are the test coding agent.", input: [{ type: "message", role: "user", content: "hello there" }], max_output_tokens: 1024 }), + }); + assert.equal(r1.status, 200); + await r1.text(); + + const s = listSessions().find((x) => x.meta.label === "img-env-sess"); + assert.ok(s); + s!.stats.lastInputTokens = STALE_BASELINE; + + process.env.BILI_IMAGE_BILLING = "bytes"; + let r2: Response; + try { + r2 = await fetch(url, { method: "POST", headers, body: imageTurn("img-env-sess") }); + } finally { + delete process.env.BILI_IMAGE_BILLING; + } + assert.equal(r2.status, 502, "env override downgrades the pixels route back to conservative byte billing"); + const err2 = JSON.parse(await r2.text()) as { error?: { code?: string } }; + assert.equal(err2.error?.code, "preflight_compress_failed"); + assert.equal(stats.streamingForwards, 1); + assert.equal(stats.imagesSeen, 0); + } finally { + proxy.close(); + await once(proxy, "close"); + upstream.close(); + await once(upstream, "close"); + } +}); + +test("e2e #767: per-route imageBilling applies under a path-qualified provider key at every gate", async () => { + const { server: upstream, port: upstreamPort, stats } = await startMockUpstream(); + const { proxy, port: proxyPort } = await startProxy(upstreamPort, { imageBilling: "pixels" }, `http://127.0.0.1:${upstreamPort}/v1`); + try { + const url = `http://127.0.0.1:${proxyPort}/bili/http://127.0.0.1:${upstreamPort}/v1/responses`; + const headers = { "content-type": "application/json" }; + + const r1 = await fetch(url, { + method: "POST", + headers, + body: JSON.stringify({ model: "gpt-astra", stream: true, store: false, session_id: "img-path-sess", instructions: "You are the test coding agent.", input: [{ type: "message", role: "user", content: "hello there" }], max_output_tokens: 1024 }), + }); + assert.equal(r1.status, 200); + await r1.text(); + + const s = listSessions().find((x) => x.meta.label === "img-path-sess"); + assert.ok(s, "session established"); + s!.stats.lastInputTokens = STALE_BASELINE; + + // Regression pin: origin-only route lookups missed this path-qualified key at the + // preflight/clamp/guard gates (auto → bytes on a bare IP host → 502); every other + // per-route setting resolves against route.rewrittenUrl instead. + const r2 = await fetch(url, { method: "POST", headers, body: imageTurn("img-path-sess") }); + assert.equal(r2.status, 200, "path-qualified per-route override reaches the preflight gate"); + await r2.text(); + assert.equal(stats.streamingForwards, 2); + assert.equal(stats.imagesSeen, 2); + + const after = listSessions().find((x) => x.meta.label === "img-path-sess")!; + assert.ok(after.stats.lastInputTokens > 0 && after.stats.lastInputTokens < WINDOW, `baseline recovered to real usage (got ${after.stats.lastInputTokens})`); + } finally { + proxy.close(); + await once(proxy, "close"); + upstream.close(); + await once(upstream, "close"); + } +});