Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions reference/request-body-budget.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,10 +10,10 @@ HTTP model requests have three separate byte boundaries:

Only two requests whose decoded body crosses 100 MiB may be active in one proxy process. A third receives 503 `request_body_busy`, `stage: decode`, `Retry-After: 1`, without being forwarded. Its slot is released on completion or cancellation. Ordinary requests do not use these slots. The limit bounds the **additional large decoded payloads**, not total process RSS: buffers, JSON parsing, session state and ordinary concurrent requests also consume memory. Streaming codecs stop on cancellation and enforce the cap while decoding; synchronous zstd can only observe cancellation at its synchronous checkpoints.

This permits a gzip / zstd / Brotli-compressed history slightly above 100 MiB to reach existing explicit reductions. It does not enable `compress.stripImages` or image compression. If the rebuilt body remains over 100 MiB, it is refused; retries cannot bypass the send guard. Already-started SSE responses deliver the failure through the existing error channel instead of changing committed headers. Budget diagnostics report sizes/stages, without request text or pixels.
This permits a gzip / zstd / Brotli-compressed history slightly above 100 MiB to cross the receive boundary and reach the rebuild pipeline. The byte budget itself performs no reduction: `stripImages` was removed (#2688) — image mass is now reduced by default media folding (#2640), which replaces large image payloads with placeholders the model can restore via `decompress({imageRef})`. If the rebuilt body is still over 100 MiB, the forward-stage 413 tells the user to fold image-bearing history first; retries cannot bypass the send guard. Already-started SSE responses deliver the failure through the existing error channel instead of changing committed headers. Budget diagnostics report sizes/stages, without request text or pixels.

Configure the existing `compress.stripImages` / `compress.stripImagesKeepRecent` policy only when historical-image removal is appropriate. The recent setting counts **messages**, not individual images. Existing image-archive / retrieval behavior, including its failure limitations, is unchanged; this is not a lossless-image lifecycle repair. Passthrough-marked or unrecognized compressed requests keep their original bytes and the raw 100 MiB limit. Uncompressed HTTP bodies above 100 MiB and WebSocket limits are unchanged. Byte limits do not replace model token-window checks or establish an upstream's accepted maximum.
There is no byte-budget policy knob left to configure for historical-image removal. Existing image-archive / retrieval behavior, including its failure limitations, is unchanged; this is not a lossless-image lifecycle repair. Passthrough-marked or unrecognized compressed requests keep their original bytes and the raw 100 MiB limit. Uncompressed HTTP bodies above 100 MiB and WebSocket limits are unchanged. Byte limits do not replace model token-window checks or establish an upstream's accepted maximum.

HTTP 请求分成三道字节边界:原始接收 100 MiB;已识别模型协议每层解压最多 200 MiB;重建后发送正文最多 100 MiB。只有两个超过旧解压上限的请求能同时占用名额,第三个在解压阶段返回可重试的 503,完成或取消后释放。该名额限制额外超大正文,不是整个进程的 RSS 上限。

此修复让略超旧限制的压缩历史进入已有、显式开启的减量流程,不会自动删除图片或开启缩图。减量后仍过大时,在正式上游发送前拒绝;流已经开始时使用现有错误事件。历史图片保留策略仍由 `stripImages` 与 `stripImagesKeepRecent` 决定,后者计数单位是消息。原图归档失败等既有生命周期问题另行处理,不能把本修复视为无损恢复保证。未压缩大正文、WebSocket、透传行为和模型 token 窗口保持原有边界。
此修复让略超旧限制的压缩历史越过接收边界进入重建管线。字节预算本身不做任何减量:`stripImages` 已移除(#2688),图片体量改由默认媒体折叠(#2640)处理——大图片被替换为占位符,模型可用 `decompress({imageRef})` 取回原图。重建后仍超限时,forward 阶段返回 413,提示先折叠含图历史;重试不能绕过发送守卫。流已经开始时使用现有错误事件。原图归档失败等既有生命周期问题另行处理,不能把本修复视为无损恢复保证。未压缩大正文、WebSocket、透传行为和模型 token 窗口保持原有边界。
40 changes: 20 additions & 20 deletions tests/image-request-budget.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ async function fixture(run: (base: string, opts: ProxyOptions, calls: Buffer[])
},
modelContextLimit: 400_000,
kernelConfig: defaultConfig(400_000),
compress: { injectTool: true, injectNudge: true, stripImages: true, stripImagesKeepRecent: 1 },
compress: { injectTool: true, injectNudge: true },
promptCache: { routing: "auto" },
sessionHeader: "x-acp-session",
log: false,
Expand Down Expand Up @@ -90,26 +90,23 @@ async function send(base: string, body: Buffer, session: string, headers: Record
return fetch(`${base}/responses`, { method: "POST", headers: { "content-type": "application/json", "content-encoding": "gzip", "session-id": session, ...headers }, body: new Uint8Array(body) });
}

test("gzip history above 100 MiB reaches explicit image reduction and preserves recent images, including tool screenshots", async () => {
test("over-budget rebuilt bodies stop at the forward-stage byte budget without touching upstream, whether images sit in messages or tool outputs", async () => {
await fixture(async (base, _opts, calls) => {
for (const tool of [false, true]) {
const r = await send(base, largeBody(tool), `large-strip-${tool}`);
assert.equal(r.status, 200, await r.text());
const body = JSON.parse(calls.at(-1)!.toString("utf8")) as { input: unknown[] };
const text = JSON.stringify(body.input);
assert.equal(text.split('"input_image"').length - 1, 1);
assert.ok(text.includes("old screenshot"));
assert.ok(text.includes(recentImage.image_url));
assert.ok(calls.at(-1)!.length < MAX_REQUEST_BYTES);
const r = await send(base, largeBody(tool), `large-forward-${tool}`);
assert.equal(r.status, 413);
const error = await r.json() as { error: { stage: string; type: string; message: string } };
assert.equal(error.error.stage, "forward");
assert.equal(error.error.type, "request_too_large");
assert.match(error.error.message, /reduce historical images or content before retrying/);
}
assert.equal(calls.length, 2);
assert.equal(calls.length, 0);
});
});

test("disabled stripping never silently drops images: rebuilt over-budget requests stop with a forward-stage 413", async () => {
await fixture(async (base, opts, calls) => {
opts.compress.stripImages = false;
const r = await send(base, largeBody(false), "large-no-strip");
test("rebuilt over-budget requests stop with a forward-stage 413 without contacting upstream", async () => {
await fixture(async (base, _opts, calls) => {
const r = await send(base, largeBody(false), "large-over-budget");
assert.equal(r.status, 413);
const error = await r.json() as { error: { stage: string; type: string } };
assert.equal(error.error.stage, "forward");
Expand All @@ -118,13 +115,13 @@ test("disabled stripping never silently drops images: rebuilt over-budget reques
});
});

test("third large decode gets retryable admission error, then succeeds after a slot is released", async () => {
test("third large decode gets retryable admission error, then hits the forward-stage budget once a slot is released", async () => {
await fixture(async (base, _opts, calls) => {
const a = new DecodedRequestAdmission(), b = new DecodedRequestAdmission();
const compressed = largeBody(false);
a.observe(MAX_REQUEST_BYTES + 1); b.observe(MAX_REQUEST_BYTES + 1);
try {
const r = await send(base, compressed, "busy-strip");
const r = await send(base, compressed, "busy-large");
assert.equal(r.status, 503);
assert.equal(r.headers.get("retry-after"), "1");
assert.equal((await r.json() as { error: { stage: string } }).error.stage, "decode");
Expand All @@ -133,9 +130,12 @@ test("third large decode gets retryable admission error, then succeeds after a s
assert.equal(small.status, 200, await small.text());
assert.equal(calls.length, 1, "normal traffic must not use the large-body slots");
a.release();
const ok = await send(base, compressed, "busy-strip-retry");
assert.equal(ok.status, 200, await ok.text());
assert.equal(calls.length, 2);
const again = await send(base, compressed, "busy-large-retry");
assert.equal(again.status, 413);
const error = await again.json() as { error: { stage: string; type: string } };
assert.equal(error.error.stage, "forward");
assert.equal(error.error.type, "request_too_large");
assert.equal(calls.length, 1, "the over-budget retry must not reach upstream either");
} finally { a.release(); b.release(); }
});
});
Expand Down
Loading