Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,10 @@ Versions follow the merge of a `*_release-v*` branch; CI publishes to npm on tag

### Fixes

- **Forward-once-then-learn for image-dominated payloads — no false 502 on pixel-tile upstreams (#496)**: the default per-image estimate (`base64 length / 4`, uncapped) matches byte-billing relays but overestimates pixel-tile upstreams (official Anthropic/OpenAI) by up to ~200×, so a session whose *estimated* image floor alone exceeded the window was hard-failed with a 502 `preflight_compress_failed` ("Images alone account for ~N tokens") even though the real cost was a few K tokens — a regression vs master for official-API multimodal users (e.g. `bili claude` pasting screenshots). The fit gate now forwards ONCE instead of hard-failing when the over-window is attributable solely to the image estimate (`textEstimate < limit`), there is no upstream evidence of overflow yet (`lastInputTokens < limit` and no learned limit for the model), and images are present. The upstream then arbitrates billing: a pixel-tile upstream accepts (usage reports small → zero behavior change); a byte-billing relay rejects once, the existing self-heal learns the true window, and every subsequent request fails fast with an accurate message — exactly one rejected forward, strictly better than master's infinite 400 loop, no new knob. Also documents `BILI_IMAGE_TOKEN_CAP` (per-image estimate cap) in CONFIGURATION.

- **Multimodal sessions: count image bytes in the preflight size gate + stop growing summaries on the Responses wire (#488)**: the preflight fit gate counted only text tokens (`estimateCoreMessages`) while the wire carried base64 images, so a session whose images alone exceeded the window was never caught by the gate, forwarded raw, rejected with `context_window_exceeded`, and self-healed into a learned window smaller than the images' cost — then every later request failed preflight fast and could never forward again (the #488 "infinite 400 loop"). New `src/image-tokens.ts` estimates per-request image tokens (inline `data:` = `base64 length / 4`; remote URLs = flat 4096) across all three wires (Responses `input_image`, OpenAI `image_url`, Anthropic `image`) and feeds them into the fit gate as `imageFloor` and into the output clamp as `imageTokens`; the overflow self-heal now folds the rejected request's image cost into the learned window so a byte-billing relay converges in one rejection instead of looping. The Responses summary payload also gained `store:false` so the proxy's ephemeral compress calls are not persisted upstream.

- **Plugin-mode passthrough strips render-tag echoes on the OpenAI chat-completions and Anthropic wires (issue #14)**: the #206/#295 tag-echo protection only ran on the Responses wire (`pipePluginResponsesWithStrip`) — plugin-mode sessions on `/v1/chat/completions` (pi, omp chat models, dsh) or `/v1/messages` (claude-code via MCP register) received the upstream stream verbatim, so a model that imitates the `<acp …>` render tags in its prose had those echoes land in the agent's replayed history, where they were re-sent to the model, re-imitated, and amplified — observed as the "bili 总是循环 / endless blank output" loop with pi + qwen3.8 (one stored assistant message carried 77 echoed tags + ~3,300 blank lines; the tags render invisibly in the pi UI, so the screen just scrolls blank). Plugin passthrough now routes chat/anthropic streams through `pipePluginChatWithStrip`: events stay byte-identical (the agent's native tool loop still sees tool calls untouched) except `choices[].delta.{content,reasoning_content,reasoning}` / `content_block_delta` `text_delta`/`thinking_delta` text, which streams through the same tag-echo state machine (one filter per field×choice/block index; held tails flush before [DONE]/message_delta/finish as synthetic deltas; echo-only chunks are dropped instead of forwarding empty deltas), and non-streaming JSON bodies strip `choices[].message.*` / `content[].{text,thinking}` the same way `pipePluginJson` already did for Responses. A `[tag-echo]` warn is logged per stripped stream for attribution.

- **Upstream proxy: unset mode now means "direct", not env auto-detect (#346)**: the web UI advertises 直连(默认) for an unset `upstreamProxyMode` (the radio defaults to `direct`) and ZCode's own default is `mode:"direct"`, but the code parsed an unset mode as "no preference" and fell through to env/system proxy auto-detection (`HTTPS_PROXY`/`HTTP_PROXY`/`ALL_PROXY`, then the Windows system proxy). A user who never configured a proxy could therefore silently have bili route the upstream through whatever proxy was in the environment (e.g. Clash), changing the egress IP — which breaks IP-bound OAuth on domestic endpoints (the #346 ZCode 401/slow symptom) and contradicts the UI. Unset mode now resolves to `direct` (an empty global proxy short-circuits to direct instead of env discovery). To follow the system/env proxy, set mode `auto` explicitly (web UI 自动(跟随系统) or `BILI_UPSTREAM_PROXY_MODE=auto`); an explicit proxy (`-F` / `BILI_UPSTREAM_PROXY`) still wins regardless of mode. (A top-level `proxy` config still requires mode `auto`/`manual` to take effect — pre-existing, unchanged.) A deduped `[upstream-proxy] <host> via <proxy> (source=<source>)` log line (non-public hosts masked per #255) now makes the resolved proxy visible instead of silent.
Expand Down
1 change: 1 addition & 0 deletions CONFIGURATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -337,6 +337,7 @@ Environment variables take precedence over the config file. They are useful for
| `ACP_COMPRESS_TOOL` | Set to `0` to disable tool injection (same as `"compress.injectTool": false`). |
| `ACP_COMPRESS_NUDGE` | Set to `0` to disable nudge injection (same as `"compress.injectNudge": false`). |
| `ACP_MODEL_CONTEXT_LIMIT` | Override the context limit globally (absolute token count). |
| `BILI_IMAGE_TOKEN_CAP` | Cap the per-image token estimate used by the preflight size gate and output clamp (#488/#496). By default an inline `data:` image counts as `base64 length / 4` tokens with **no cap** — correct for byte-billing relays, but a large over-estimate for pixel-tile upstreams (official Anthropic/OpenAI), which bill each image at roughly 1.1K–1.6K tokens regardless of byte size. Set this to your upstream's per-image tile cost so the gate reflects real billing; unset = no cap (current default). |
| `BILI_CONFIG_FILE` | Override the config file path (point at any JSON file). |
| `ACP_PORT` / `PORT` | Override the listen port. |
| `ACP_HOST` | Override the listen host. |
Expand Down
1 change: 1 addition & 0 deletions CONFIGURATION.zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -335,6 +335,7 @@
| `ACP_COMPRESS_TOOL` | 设为 `0` 禁用工具注入(等同 `"compress.injectTool": false`)。 |
| `ACP_COMPRESS_NUDGE` | 设为 `0` 禁用 nudge 注入(等同 `"compress.injectNudge": false`)。 |
| `ACP_MODEL_CONTEXT_LIMIT` | 全局覆盖上下文上限(绝对 token 数)。 |
| `BILI_IMAGE_TOKEN_CAP` | 预检尺寸门与输出钳制用的单图 token 估算上限(#488/#496)。默认内联 `data:` 图片按 `base64 长度 / 4` 计 token、**无上限** —— 对字节计费 relay 正确,但对像素 tile 计费的官方上游(Anthropic/OpenAI)会严重高估(后者无论字节多少,每图约计 1.1K–1.6K token)。设为你上游的单图 tile 成本,可让尺寸门反映真实计费;不设置 = 无上限(当前默认)。 |
| `BILI_CONFIG_FILE` | 覆盖配置文件路径(指向任意 JSON 文件)。 |
| `ACP_PORT` / `PORT` | 覆盖监听端口。 |
| `ACP_HOST` | 覆盖监听主机。 |
Expand Down
22 changes: 21 additions & 1 deletion src/server.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2298,9 +2298,29 @@ async function preflightCompressIfNeeded(
// #488: images are forwarded verbatim but invisible to the kernel's text model —
// add their cost to every size decision here (trigger, fit gates, self-heal).
const imageTokens = imageTokensInRawBody(prepared.protocol, prepared.body);
const payloadEstimate = estimateCoreMessages(prepared.processedMessages) + imageTokens;
const textEstimate = estimateCoreMessages(prepared.processedMessages);
const payloadEstimate = textEstimate + imageTokens;
const tokenCount = Math.max(session.stats.lastInputTokens, payloadEstimate);
if (limit <= 0 || !model || tokenCount < limit) return prepared;
// #496 forward-once-then-learn: the default image cost (base64/4) matches byte
// relays (#488) but overestimates pixel-tile upstreams (a 400KB JPEG ≈ 1.6K real
// tokens, not ~133K), so an image-dominated payload can clear the window on ESTIMATE
// alone. When images are the sole over-window component (text fits) and we hold no
// upstream overflow evidence (measured baseline under window + no learned limit for
// this model), forward once and let the upstream arbitrate billing: tile upstreams
// accept it; byte relays reject it (400) → forward()'s self-heal learns the window
// (it counts rejected image tokens) → later requests fail-fast. #488's 400 loop stays
// broken (exactly one rejected forward). With either evidence signal present we trust
// the estimate and fall through to fold / fail-fast below.
const learnedMap = session.metadata.learnedContextLimits as Record<string, number> | undefined;
const learnedLimit =
(model ? learnedMap?.[model] : undefined) ??
(session.metadata.learnedContextLimit as number | undefined);
const noOverflowEvidence = session.stats.lastInputTokens < limit && learnedLimit === undefined;
if (imageTokens > 0 && textEstimate < limit && noOverflowEvidence) {
log("warn", `[${session.id}] image-dominated payload (~${textEstimate} text + ~${imageTokens} image tokens) exceeds window ${limit} by estimate only, no upstream overflow evidence — forwarding once so the upstream arbitrates billing (#496)`);
return prepared;
}
// #301: forwarding as-is is safe ONLY when the payload's own estimate
// fits the window. The trigger (and the loop's fit check) floor on
// session.stats.lastInputTokens, which can be stale — e.g. a
Expand Down
42 changes: 33 additions & 9 deletions tests/fix-488-multimodal.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -165,16 +165,19 @@ test("e2e #488-B (Responses): preflight summary requests carry store:false so co
}
});

test("e2e #488-A (Responses): images alone over the window are withheld with an actionable error, never forwarded", async () => {
test("e2e #488-A/#496 (Responses): image-dominated payload with no overflow evidence is forwarded once; tile upstream accepts", async () => {
const streamForwards: boolean[] = [];
const upstream = http.createServer((req, res) => {
const chunks: Buffer[] = [];
req.on("data", (c: Buffer) => chunks.push(c));
req.on("end", () => {
const parsed = JSON.parse(Buffer.concat(chunks).toString("utf8")) as { stream?: boolean };
streamForwards.push(parsed.stream === true);
// Pixel-tile upstream shape: accepts the payload and reports a SMALL real
// input cost despite the large base64 body (a 60k-char screenshot bills as
// ~1.6K tiles upstream, not the 15K tokens bili estimates from b64/4).
res.writeHead(200, { "content-type": "text/event-stream", "cache-control": "no-cache" });
res.write(completed(100));
res.write(completed(3000));
res.end();
});
});
Expand All @@ -184,8 +187,13 @@ test("e2e #488-A (Responses): images alone over the window are withheld with an
const { proxy, url } = await startProxy(upstreamPort);

try {
// 7 screenshots × 60k base64 chars = 7 × 15_000 = 105_000 image tokens,
// against a 10_000 window — no amount of text folding can fix this.
// 7 screenshots × 60k base64 chars = 7 × 15_000 = 105_000 ESTIMATED image
// tokens against a 10_000 window. Text is trivially small, so images are the
// sole over-window component. Fresh session (lastInputTokens=0, no learned
// limit) → no upstream overflow evidence → #496 forward-once-then-learn lets
// the upstream arbitrate the true billing instead of a hard 502 (#488's
// original expectation was withheld+never-forwarded; that was the false
// positive for tile-billed upstreams this issue fixes).
const bigB64 = "A".repeat(60_000);
const input = [
{
Expand All @@ -202,11 +210,27 @@ test("e2e #488-A (Responses): images alone over the window are withheld with an
headers: { "content-type": "application/json" },
body: JSON.stringify({ model: "gpt-resp", stream: true, session_id: "img-sess-a", instructions: "You are the test coding agent.", input }),
});
assert.equal(r.status, 502, "over-window multimodal payload is withheld");
const err = JSON.parse(await r.text()) as { error?: { code?: string; message?: string; retryable?: boolean } };
assert.equal(err.error?.code, "preflight_compress_failed");
assert.match(err.error?.message ?? "", /Images alone account for ~105000 tokens/);
assert.ok(!streamForwards.includes(true), "the over-window payload was never forwarded upstream");
assert.equal(r.status, 200, "tile-billed upstream accepts the forwarded payload (#496: no hard 502)");
await r.text();
assert.ok(streamForwards.includes(true), "the image-dominated payload was forwarded once for the upstream to arbitrate");

// Steady state: a follow-up turn on the same session also forwards and succeeds
// (real usage stays under the window), so multimodal sessions keep working.
const r2 = await fetch(url, {
method: "POST",
headers: { "content-type": "application/json" },
body: JSON.stringify({
model: "gpt-resp", stream: true, session_id: "img-sess-a", instructions: "You are the test coding agent.",
input: [
...input,
{ type: "message", role: "assistant", content: "I see the failing screen." },
{ type: "message", role: "user", content: [{ type: "input_text", text: "and one more screenshot" }, { type: "input_image", image_url: `data:image/png;base64,${bigB64}` }] },
],
}),
});
assert.equal(r2.status, 200, "follow-up multimodal turn still succeeds (no fail-fast regression)");
await r2.text();
assert.ok(streamForwards.filter(Boolean).length >= 2, "both turns were forwarded");
} finally {
await closeAll(proxy, upstream);
}
Expand Down
Loading
Loading