diff --git a/CHANGELOG.md b/CHANGELOG.md index 27d06ad7c8..cf946b79ad 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,10 @@ Versions follow the merge of a `*_release-v*` branch; CI publishes to npm on tag ### Fixes +- **Forward-once-then-learn for image-dominated payloads — no false 502 on pixel-tile upstreams (#496)**: the default per-image estimate (`base64 length / 4`, uncapped) matches byte-billing relays but overestimates pixel-tile upstreams (official Anthropic/OpenAI) by up to ~200×, so a session whose *estimated* image floor alone exceeded the window was hard-failed with a 502 `preflight_compress_failed` ("Images alone account for ~N tokens") even though the real cost was a few K tokens — a regression vs master for official-API multimodal users (e.g. `bili claude` pasting screenshots). The fit gate now forwards ONCE instead of hard-failing when the over-window is attributable solely to the image estimate (`textEstimate < limit`), there is no upstream evidence of overflow yet (`lastInputTokens < limit` and no learned limit for the model), and images are present. The upstream then arbitrates billing: a pixel-tile upstream accepts (usage reports small → zero behavior change); a byte-billing relay rejects once, the existing self-heal learns the true window, and every subsequent request fails fast with an accurate message — exactly one rejected forward, strictly better than master's infinite 400 loop, no new knob. Also documents `BILI_IMAGE_TOKEN_CAP` (per-image estimate cap) in CONFIGURATION. + +- **Multimodal sessions: count image bytes in the preflight size gate + stop growing summaries on the Responses wire (#488)**: the preflight fit gate counted only text tokens (`estimateCoreMessages`) while the wire carried base64 images, so a session whose images alone exceeded the window was never caught by the gate, forwarded raw, rejected with `context_window_exceeded`, and self-healed into a learned window smaller than the images' cost — then every later request failed preflight fast and could never forward again (the #488 "infinite 400 loop"). New `src/image-tokens.ts` estimates per-request image tokens (inline `data:` = `base64 length / 4`; remote URLs = flat 4096) across all three wires (Responses `input_image`, OpenAI `image_url`, Anthropic `image`) and feeds them into the fit gate as `imageFloor` and into the output clamp as `imageTokens`; the overflow self-heal now folds the rejected request's image cost into the learned window so a byte-billing relay converges in one rejection instead of looping. The Responses summary payload also gained `store:false` so the proxy's ephemeral compress calls are not persisted upstream. + - **Plugin-mode passthrough strips render-tag echoes on the OpenAI chat-completions and Anthropic wires (issue #14)**: the #206/#295 tag-echo protection only ran on the Responses wire (`pipePluginResponsesWithStrip`) — plugin-mode sessions on `/v1/chat/completions` (pi, omp chat models, dsh) or `/v1/messages` (claude-code via MCP register) received the upstream stream verbatim, so a model that imitates the `` render tags in its prose had those echoes land in the agent's replayed history, where they were re-sent to the model, re-imitated, and amplified — observed as the "bili 总是循环 / endless blank output" loop with pi + qwen3.8 (one stored assistant message carried 77 echoed tags + ~3,300 blank lines; the tags render invisibly in the pi UI, so the screen just scrolls blank). Plugin passthrough now routes chat/anthropic streams through `pipePluginChatWithStrip`: events stay byte-identical (the agent's native tool loop still sees tool calls untouched) except `choices[].delta.{content,reasoning_content,reasoning}` / `content_block_delta` `text_delta`/`thinking_delta` text, which streams through the same tag-echo state machine (one filter per field×choice/block index; held tails flush before [DONE]/message_delta/finish as synthetic deltas; echo-only chunks are dropped instead of forwarding empty deltas), and non-streaming JSON bodies strip `choices[].message.*` / `content[].{text,thinking}` the same way `pipePluginJson` already did for Responses. A `[tag-echo]` warn is logged per stripped stream for attribution. - **Upstream proxy: unset mode now means "direct", not env auto-detect (#346)**: the web UI advertises 直连(默认) for an unset `upstreamProxyMode` (the radio defaults to `direct`) and ZCode's own default is `mode:"direct"`, but the code parsed an unset mode as "no preference" and fell through to env/system proxy auto-detection (`HTTPS_PROXY`/`HTTP_PROXY`/`ALL_PROXY`, then the Windows system proxy). A user who never configured a proxy could therefore silently have bili route the upstream through whatever proxy was in the environment (e.g. Clash), changing the egress IP — which breaks IP-bound OAuth on domestic endpoints (the #346 ZCode 401/slow symptom) and contradicts the UI. Unset mode now resolves to `direct` (an empty global proxy short-circuits to direct instead of env discovery). To follow the system/env proxy, set mode `auto` explicitly (web UI 自动(跟随系统) or `BILI_UPSTREAM_PROXY_MODE=auto`); an explicit proxy (`-F` / `BILI_UPSTREAM_PROXY`) still wins regardless of mode. (A top-level `proxy` config still requires mode `auto`/`manual` to take effect — pre-existing, unchanged.) A deduped `[upstream-proxy] via (source=)` log line (non-public hosts masked per #255) now makes the resolved proxy visible instead of silent. diff --git a/CONFIGURATION.md b/CONFIGURATION.md index 35e11064ec..5021c2b3af 100644 --- a/CONFIGURATION.md +++ b/CONFIGURATION.md @@ -337,6 +337,7 @@ Environment variables take precedence over the config file. They are useful for | `ACP_COMPRESS_TOOL` | Set to `0` to disable tool injection (same as `"compress.injectTool": false`). | | `ACP_COMPRESS_NUDGE` | Set to `0` to disable nudge injection (same as `"compress.injectNudge": false`). | | `ACP_MODEL_CONTEXT_LIMIT` | Override the context limit globally (absolute token count). | +| `BILI_IMAGE_TOKEN_CAP` | Cap the per-image token estimate used by the preflight size gate and output clamp (#488/#496). By default an inline `data:` image counts as `base64 length / 4` tokens with **no cap** — correct for byte-billing relays, but a large over-estimate for pixel-tile upstreams (official Anthropic/OpenAI), which bill each image at roughly 1.1K–1.6K tokens regardless of byte size. Set this to your upstream's per-image tile cost so the gate reflects real billing; unset = no cap (current default). | | `BILI_CONFIG_FILE` | Override the config file path (point at any JSON file). | | `ACP_PORT` / `PORT` | Override the listen port. | | `ACP_HOST` | Override the listen host. | diff --git a/CONFIGURATION.zh-CN.md b/CONFIGURATION.zh-CN.md index 234a82e4a5..c950378fce 100644 --- a/CONFIGURATION.zh-CN.md +++ b/CONFIGURATION.zh-CN.md @@ -335,6 +335,7 @@ | `ACP_COMPRESS_TOOL` | 设为 `0` 禁用工具注入(等同 `"compress.injectTool": false`)。 | | `ACP_COMPRESS_NUDGE` | 设为 `0` 禁用 nudge 注入(等同 `"compress.injectNudge": false`)。 | | `ACP_MODEL_CONTEXT_LIMIT` | 全局覆盖上下文上限(绝对 token 数)。 | +| `BILI_IMAGE_TOKEN_CAP` | 预检尺寸门与输出钳制用的单图 token 估算上限(#488/#496)。默认内联 `data:` 图片按 `base64 长度 / 4` 计 token、**无上限** —— 对字节计费 relay 正确,但对像素 tile 计费的官方上游(Anthropic/OpenAI)会严重高估(后者无论字节多少,每图约计 1.1K–1.6K token)。设为你上游的单图 tile 成本,可让尺寸门反映真实计费;不设置 = 无上限(当前默认)。 | | `BILI_CONFIG_FILE` | 覆盖配置文件路径(指向任意 JSON 文件)。 | | `ACP_PORT` / `PORT` | 覆盖监听端口。 | | `ACP_HOST` | 覆盖监听主机。 | diff --git a/src/server.ts b/src/server.ts index 941bbd5b9d..642c37807f 100644 --- a/src/server.ts +++ b/src/server.ts @@ -2298,9 +2298,29 @@ async function preflightCompressIfNeeded( // #488: images are forwarded verbatim but invisible to the kernel's text model — // add their cost to every size decision here (trigger, fit gates, self-heal). const imageTokens = imageTokensInRawBody(prepared.protocol, prepared.body); - const payloadEstimate = estimateCoreMessages(prepared.processedMessages) + imageTokens; + const textEstimate = estimateCoreMessages(prepared.processedMessages); + const payloadEstimate = textEstimate + imageTokens; const tokenCount = Math.max(session.stats.lastInputTokens, payloadEstimate); if (limit <= 0 || !model || tokenCount < limit) return prepared; + // #496 forward-once-then-learn: the default image cost (base64/4) matches byte + // relays (#488) but overestimates pixel-tile upstreams (a 400KB JPEG ≈ 1.6K real + // tokens, not ~133K), so an image-dominated payload can clear the window on ESTIMATE + // alone. When images are the sole over-window component (text fits) and we hold no + // upstream overflow evidence (measured baseline under window + no learned limit for + // this model), forward once and let the upstream arbitrate billing: tile upstreams + // accept it; byte relays reject it (400) → forward()'s self-heal learns the window + // (it counts rejected image tokens) → later requests fail-fast. #488's 400 loop stays + // broken (exactly one rejected forward). With either evidence signal present we trust + // the estimate and fall through to fold / fail-fast below. + const learnedMap = session.metadata.learnedContextLimits as Record | undefined; + const learnedLimit = + (model ? learnedMap?.[model] : undefined) ?? + (session.metadata.learnedContextLimit as number | undefined); + const noOverflowEvidence = session.stats.lastInputTokens < limit && learnedLimit === undefined; + if (imageTokens > 0 && textEstimate < limit && noOverflowEvidence) { + log("warn", `[${session.id}] image-dominated payload (~${textEstimate} text + ~${imageTokens} image tokens) exceeds window ${limit} by estimate only, no upstream overflow evidence — forwarding once so the upstream arbitrates billing (#496)`); + return prepared; + } // #301: forwarding as-is is safe ONLY when the payload's own estimate // fits the window. The trigger (and the loop's fit check) floor on // session.stats.lastInputTokens, which can be stale — e.g. a diff --git a/tests/fix-488-multimodal.test.ts b/tests/fix-488-multimodal.test.ts index e45626d23c..63cb143a7e 100644 --- a/tests/fix-488-multimodal.test.ts +++ b/tests/fix-488-multimodal.test.ts @@ -165,7 +165,7 @@ test("e2e #488-B (Responses): preflight summary requests carry store:false so co } }); -test("e2e #488-A (Responses): images alone over the window are withheld with an actionable error, never forwarded", async () => { +test("e2e #488-A/#496 (Responses): image-dominated payload with no overflow evidence is forwarded once; tile upstream accepts", async () => { const streamForwards: boolean[] = []; const upstream = http.createServer((req, res) => { const chunks: Buffer[] = []; @@ -173,8 +173,11 @@ test("e2e #488-A (Responses): images alone over the window are withheld with an req.on("end", () => { const parsed = JSON.parse(Buffer.concat(chunks).toString("utf8")) as { stream?: boolean }; streamForwards.push(parsed.stream === true); + // Pixel-tile upstream shape: accepts the payload and reports a SMALL real + // input cost despite the large base64 body (a 60k-char screenshot bills as + // ~1.6K tiles upstream, not the 15K tokens bili estimates from b64/4). res.writeHead(200, { "content-type": "text/event-stream", "cache-control": "no-cache" }); - res.write(completed(100)); + res.write(completed(3000)); res.end(); }); }); @@ -184,8 +187,13 @@ test("e2e #488-A (Responses): images alone over the window are withheld with an const { proxy, url } = await startProxy(upstreamPort); try { - // 7 screenshots × 60k base64 chars = 7 × 15_000 = 105_000 image tokens, - // against a 10_000 window — no amount of text folding can fix this. + // 7 screenshots × 60k base64 chars = 7 × 15_000 = 105_000 ESTIMATED image + // tokens against a 10_000 window. Text is trivially small, so images are the + // sole over-window component. Fresh session (lastInputTokens=0, no learned + // limit) → no upstream overflow evidence → #496 forward-once-then-learn lets + // the upstream arbitrate the true billing instead of a hard 502 (#488's + // original expectation was withheld+never-forwarded; that was the false + // positive for tile-billed upstreams this issue fixes). const bigB64 = "A".repeat(60_000); const input = [ { @@ -202,11 +210,27 @@ test("e2e #488-A (Responses): images alone over the window are withheld with an headers: { "content-type": "application/json" }, body: JSON.stringify({ model: "gpt-resp", stream: true, session_id: "img-sess-a", instructions: "You are the test coding agent.", input }), }); - assert.equal(r.status, 502, "over-window multimodal payload is withheld"); - const err = JSON.parse(await r.text()) as { error?: { code?: string; message?: string; retryable?: boolean } }; - assert.equal(err.error?.code, "preflight_compress_failed"); - assert.match(err.error?.message ?? "", /Images alone account for ~105000 tokens/); - assert.ok(!streamForwards.includes(true), "the over-window payload was never forwarded upstream"); + assert.equal(r.status, 200, "tile-billed upstream accepts the forwarded payload (#496: no hard 502)"); + await r.text(); + assert.ok(streamForwards.includes(true), "the image-dominated payload was forwarded once for the upstream to arbitrate"); + + // Steady state: a follow-up turn on the same session also forwards and succeeds + // (real usage stays under the window), so multimodal sessions keep working. + const r2 = await fetch(url, { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ + model: "gpt-resp", stream: true, session_id: "img-sess-a", instructions: "You are the test coding agent.", + input: [ + ...input, + { type: "message", role: "assistant", content: "I see the failing screen." }, + { type: "message", role: "user", content: [{ type: "input_text", text: "and one more screenshot" }, { type: "input_image", image_url: `data:image/png;base64,${bigB64}` }] }, + ], + }), + }); + assert.equal(r2.status, 200, "follow-up multimodal turn still succeeds (no fail-fast regression)"); + await r2.text(); + assert.ok(streamForwards.filter(Boolean).length >= 2, "both turns were forwarded"); } finally { await closeAll(proxy, upstream); } diff --git a/tests/issue496-forward-once-learn.test.ts b/tests/issue496-forward-once-learn.test.ts new file mode 100644 index 0000000000..a9226b5a75 --- /dev/null +++ b/tests/issue496-forward-once-learn.test.ts @@ -0,0 +1,138 @@ +import assert from "node:assert/strict"; +import http from "node:http"; +import { once } from "node:events"; +import test from "node:test"; + +process.env.NODE_ENV = "test"; + +import { defaultConfig } from "acp-kernel"; +import { startServer, type ProxyOptions } from "../src/server.ts"; +import { SessionStore, _setStoreForTest } from "../src/persist.ts"; +import { _setForTest as setRegistryForTest } from "../src/registry.ts"; +import { listSessions } from "../src/session.ts"; + +// Issue #496: the default image cost (base64/4) matches byte-counting relays (#488) +// but overestimates pixel-tile upstreams, so PR #489's fit gate hard-fails (502) an +// image-dominated payload whose REAL cost is tiny. Fix = forward-once-then-learn: when +// images are the sole over-window component and there is NO upstream overflow evidence, +// forward once and let the upstream arbitrate. This file pins the OTHER billing model: +// a byte-counting relay that genuinely rejects the payload must see exactly ONE rejected +// forward, after which the self-heal learns the window and later requests fail fast — +// i.e. #488's infinite 400 loop is NOT reintroduced by the fix. + +const OVERFLOW_BODY = JSON.stringify({ + error: { type: "invalid_request_error", message: "context_window_exceeded: prompt exceeds the context window" }, +}); + +function summaryJson(): string { + return JSON.stringify({ + id: "msg_summary", + type: "message", + role: "assistant", + model: "claude-img", + content: [{ type: "text", text: "PREFLIGHT SUMMARY: folded segment." }], + stop_reason: "end_turn", + usage: { input_tokens: 500, output_tokens: 50 }, + }); +} + +function screenshotPayload(): string { + // 7 screenshots × 60k base64 chars = 7 × 15_000 = 105_000 ESTIMATED image tokens + // against a 10_000 window; the text is trivially small, so images are the sole + // over-window component (textEstimate < limit) — the forward-once trigger shape. + const bigB64 = "A".repeat(60_000); + const images = Array.from({ length: 7 }, () => ({ type: "image", source: { type: "base64", media_type: "image/png", data: bigB64 } })); + return JSON.stringify({ + model: "claude-img", + max_tokens: 1024, + stream: true, + messages: [{ role: "user", content: [{ type: "text", text: "seven screenshots attached" }, ...images] }], + }); +} + +test("e2e #496 (byte-counting relay): one rejected forward, then fail-fast — the 400 loop is not reintroduced", async () => { + const streamForwards: boolean[] = []; + const upstream = http.createServer((req, res) => { + const chunks: Buffer[] = []; + req.on("data", (c: Buffer) => chunks.push(c)); + req.on("end", () => { + const raw = Buffer.concat(chunks).toString("utf8"); + let parsed: { stream?: boolean } = {}; + try { parsed = JSON.parse(raw); } catch { parsed = {}; } + // A preflight summarization call (non-streaming) — answer it so a fold + // attempt on a later request completes instead of hanging the proxy. + if (parsed.stream === false) { + res.writeHead(200, { "content-type": "application/json" }); + res.end(summaryJson()); + return; + } + // Every streaming request is a genuine client forward: the byte-counting + // relay rejects the oversized multimodal payload with a context overflow. + streamForwards.push(true); + res.writeHead(400, { "content-type": "application/json" }); + res.end(OVERFLOW_BODY); + }); + }); + upstream.listen(0, "127.0.0.1"); + await once(upstream, "listening"); + const upstreamPort = (upstream.address() as { port: number }).port; + + _setStoreForTest(new SessionStore({ enabled: false })); + setRegistryForTest({}); + const proxy = await startServer({ + port: 0, + host: "127.0.0.1", + upstream: "http://127.0.0.1", + routes: { [`http://127.0.0.1:${upstreamPort}`]: { models: { "claude-img": { context: 10_000 } } } }, + modelContextLimit: 10_000, + kernelConfig: defaultConfig(10_000), + compress: { injectTool: true, injectNudge: true }, + promptCache: { routing: "auto" }, + sessionHeader: "x-acp-session", + log: false, + debug: false, + passthrough: false, + autoUpdate: false, + mitm: { enabled: false, domains: [] }, + } as ProxyOptions); + await once(proxy, "listening"); + const proxyPort = (proxy.address() as { port: number }).port; + + try { + const url = `http://127.0.0.1:${proxyPort}/bili/http://127.0.0.1:${upstreamPort}/v1/messages`; + const headers = { "content-type": "application/json", "x-acp-session": "img-relay-sess" }; + + // --- Request 1: no overflow evidence yet → forward ONCE; relay rejects 400 --- + const r1 = await fetch(url, { method: "POST", headers, body: screenshotPayload() }); + assert.equal(r1.status, 400, "first forward passes the relay's 400 through verbatim"); + const r1text = await r1.text(); + assert.ok(r1text.includes("context_window_exceeded"), "relay error body passes through"); + assert.deepEqual(streamForwards, [true], "request 1 was forwarded exactly once"); + + // The self-heal recognized the overflow and learned a conservative window from + // the REJECTED payload size (which counts the images, #488), arming the shrink. + const s = listSessions().find((x) => (x.metadata.learnedContextLimits as Record | undefined)?.["claude-img"] !== undefined); + assert.ok(s, "session learned a conservative window from the rejected multimodal payload"); + const learned = (s!.metadata.learnedContextLimits as Record)["claude-img"]; + assert.ok(learned >= 1000 && learned > 10_000, `learned window reflects the image-heavy payload (got ${learned})`); + assert.ok(s!.stats.lastInputTokens >= learned, "emergency shrink armed (lastInputTokens >= learned window)"); + + // --- Request 2: overflow EVIDENCE now exists → NO forward-once → fail-fast --- + const r2 = await fetch(url, { method: "POST", headers, body: screenshotPayload() }); + assert.equal(r2.status, 502, "second request fails fast instead of forwarding again"); + const err2 = JSON.parse(await r2.text()) as { error?: { code?: string } }; + assert.equal(err2.error?.code, "preflight_compress_failed"); + assert.deepEqual(streamForwards, [true], "request 2 was NEVER forwarded (loop broken after one rejection)"); + + // --- Request 3: stable — still fail-fast, still exactly one forward total --- + const r3 = await fetch(url, { method: "POST", headers, body: screenshotPayload() }); + assert.equal(r3.status, 502, "third request also fails fast"); + await r3.text(); + assert.deepEqual(streamForwards, [true], "still exactly one forward total (no re-forwarding loop)"); + } finally { + proxy.close(); + await once(proxy, "close"); + upstream.close(); + await once(upstream, "close"); + } +});