Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions reference/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,7 @@ billion-context/
│ ├── conflict-watch.ts # Per-session compression-conflict ledger (#1206)
│ ├── upstream-proxy.ts # undici ProxyAgent routing (explicit-direct vs no-preference, SOCKS reject)
│ ├── fetch-util.ts # Capped body reads, timeout/replay helpers (knob-resolved)
│ ├── request-body-budget.ts # 接收、解压、发送字节预算与大请求名额
│ ├── fetch-transport.ts # AsyncLocalStorage fetch-transport override (withFetchTransport)
│ ├── content-encoding.ts # gzip/deflate decode + decompression-bomb guard (413)
│ ├── wire-body.ts # Trailing-user-turn append on assembled wire bodies
Expand Down
19 changes: 19 additions & 0 deletions reference/request-body-budget.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
# Request body byte budgets / 请求体字节预算

HTTP model requests have three separate byte boundaries:

| Stage | Limit | Failure |
| --- | --- | --- |
| Raw HTTP reception | 100 MiB | 413, `request_too_large`, `stage: receive` when the socket permits a response |
| Decompression of recognized model protocols | 200 MiB per decoding stage | 413, `request_too_large`, `stage: decode` |
| Rebuilt upstream body | 100 MiB of UTF-8 / Buffer bytes | 413, `request_too_large`, `stage: forward` before initial send |

Only two requests whose decoded body crosses 100 MiB may be active in one proxy process. A third receives 503 `request_body_busy`, `stage: decode`, `Retry-After: 1`, without being forwarded. Its slot is released on completion or cancellation. Ordinary requests do not use these slots. The limit bounds the **additional large decoded payloads**, not total process RSS: buffers, JSON parsing, session state and ordinary concurrent requests also consume memory. Streaming codecs stop on cancellation and enforce the cap while decoding; synchronous zstd can only observe cancellation at its synchronous checkpoints.

This permits a gzip / zstd / Brotli-compressed history slightly above 100 MiB to reach existing explicit reductions. It does not enable `compress.stripImages` or image compression. If the rebuilt body remains over 100 MiB, it is refused; retries cannot bypass the send guard. Already-started SSE responses deliver the failure through the existing error channel instead of changing committed headers. Budget diagnostics report sizes/stages, without request text or pixels.

Configure the existing `compress.stripImages` / `compress.stripImagesKeepRecent` policy only when historical-image removal is appropriate. The recent setting counts **messages**, not individual images. Existing image-archive / retrieval behavior, including its failure limitations, is unchanged; this is not a lossless-image lifecycle repair. Passthrough-marked or unrecognized compressed requests keep their original bytes and the raw 100 MiB limit. Uncompressed HTTP bodies above 100 MiB and WebSocket limits are unchanged. Byte limits do not replace model token-window checks or establish an upstream's accepted maximum.

HTTP 请求分成三道字节边界:原始接收 100 MiB;已识别模型协议每层解压最多 200 MiB;重建后发送正文最多 100 MiB。只有两个超过旧解压上限的请求能同时占用名额,第三个在解压阶段返回可重试的 503,完成或取消后释放。该名额限制额外超大正文,不是整个进程的 RSS 上限。

此修复让略超旧限制的压缩历史进入已有、显式开启的减量流程,不会自动删除图片或开启缩图。减量后仍过大时,在正式上游发送前拒绝;流已经开始时使用现有错误事件。历史图片保留策略仍由 `stripImages` 与 `stripImagesKeepRecent` 决定,后者计数单位是消息。原图归档失败等既有生命周期问题另行处理,不能把本修复视为无损恢复保证。未压缩大正文、WebSocket、透传行为和模型 token 窗口保持原有边界。
50 changes: 34 additions & 16 deletions src/content-encoding.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
import { createBrotliDecompress, createGunzip, createInflate, createInflateRaw } from "node:zlib";
import { DecodedRequestBusyError } from "./request-body-budget.js";
import type { Duplex } from "node:stream";
import { Decompress as ZstdDecompress } from "fzstd";

Expand All @@ -19,48 +20,63 @@ export class DecompressedTooLargeError extends Error {
* instead reports overflow opaquely (ERR_BUFFER_TOO_LARGE for gzip/br; an
* ambiguous Z_DATA_ERROR for deflate that cannot be told apart from a corrupt
* stream), so counting the stream ourselves is the only reliable signal. */
function streamDecode(factory: () => Duplex, input: Buffer, max: number): Promise<Buffer> {
interface DecodeOptions {
signal?: AbortSignal;
onBytes?: (bytes: number) => void;
}

function streamDecode(factory: () => Duplex, input: Buffer, max: number, options?: DecodeOptions): Promise<Buffer> {
return new Promise((resolve, reject) => {
const chunks: Buffer[] = [];
let size = 0;
const decoder = factory();
const abort = () => decoder.destroy(options?.signal?.reason ?? new Error("request decoding aborted"));
decoder.once("close", () => options?.signal?.removeEventListener("abort", abort));
decoder.on("data", (chunk: Buffer) => {
if (decoder.destroyed) return;
size += chunk.length;
if (size > max) {
decoder.destroy();
reject(new DecompressedTooLargeError(max));
return;
try {
if (size > max) throw new DecompressedTooLargeError(max);
options?.onBytes?.(size);
chunks.push(chunk);
} catch (err) {
decoder.destroy(err instanceof Error ? err : new Error(String(err)));
}
chunks.push(chunk);
});
decoder.on("end", () => resolve(Buffer.concat(chunks, size)));
decoder.on("error", (err: Error) => reject(err));
decoder.end(input);
if (options?.signal?.aborted) abort();
else {
options?.signal?.addEventListener("abort", abort, { once: true });
decoder.end(input);
}
});
}

async function decodeZstd(input: Buffer, max: number): Promise<Buffer> {
async function decodeZstd(input: Buffer, max: number, options?: DecodeOptions): Promise<Buffer> {
const chunks: Buffer[] = [];
let size = 0;
const decoder = new ZstdDecompress((chunk) => {
size += chunk.byteLength;
if (size > max) throw new DecompressedTooLargeError(max);
options?.signal?.throwIfAborted();
options?.onBytes?.(size);
chunks.push(Buffer.from(chunk));
});
decoder.push(input, true);
return Buffer.concat(chunks, size);
}

async function decodeOne(coding: string, input: Buffer, max: number): Promise<Buffer> {
if (coding === "gzip" || coding === "x-gzip") return streamDecode(createGunzip, input, max);
if (coding === "br") return streamDecode(createBrotliDecompress, input, max);
if (coding === "zstd") return decodeZstd(input, max);
async function decodeOne(coding: string, input: Buffer, max: number, options?: DecodeOptions): Promise<Buffer> {
if (coding === "gzip" || coding === "x-gzip") return streamDecode(createGunzip, input, max, options);
if (coding === "br") return streamDecode(createBrotliDecompress, input, max, options);
if (coding === "zstd") return decodeZstd(input, max, options);
if (coding === "deflate") {
try {
return await streamDecode(createInflate, input, max);
return await streamDecode(createInflate, input, max, options);
} catch (err) {
if (err instanceof DecompressedTooLargeError) throw err;
return streamDecode(createInflateRaw, input, max);
if (err instanceof DecompressedTooLargeError || err instanceof DecodedRequestBusyError || options?.signal?.aborted) throw err;
return streamDecode(createInflateRaw, input, max, options);
}
}
throw new Error(`unsupported request content-encoding: ${coding}`);
Expand All @@ -70,15 +86,17 @@ export async function decodeRequestBody(
contentEncoding: string | undefined,
input: Buffer,
maxOutputBytes: number,
options?: DecodeOptions,
): Promise<{ body: Buffer; decoded: boolean }> {
options?.signal?.throwIfAborted();
const codings = (contentEncoding ?? "")
.split(",")
.map((coding) => coding.trim().toLowerCase())
.filter((coding) => coding && coding !== "identity");
if (codings.length === 0) return { body: input, decoded: false };
let body = input;
for (const coding of codings.reverse()) {
body = await decodeOne(coding, body, maxOutputBytes);
body = await decodeOne(coding, body, maxOutputBytes, options);
if (body.byteLength > maxOutputBytes) throw new DecompressedTooLargeError(maxOutputBytes);
}
return { body, decoded: true };
Expand Down
10 changes: 6 additions & 4 deletions src/fetch-util.ts
Original file line number Diff line number Diff line change
Expand Up @@ -11,14 +11,15 @@ import {
/** HTTP robustness helpers for the proxy.

- readBody is capped: an unbounded request body is a memory-exhaustion
vector when the proxy listens publicly. 100 MB is generous for LLM
payloads (which can carry large tool results / file contents) while
still rejecting pathological sizes.
vector when the proxy listens publicly. Wire intake and rebuilt sends
stay capped at 100 MiB; recognized encoded requests have a separate,
bounded decode allowance before existing reductions run.
- fetchWithTimeout wraps upstream requests with an AbortController so a
stuck upstream cannot hold a client connection open forever. LLM
streams can legitimately run for minutes, so the default is long. */

export const MAX_REQUEST_BYTES = 100 * 1024 * 1024;
export { MAX_REQUEST_BYTES } from "./request-body-budget.js";
import { checkOutboundBody } from "./request-body-budget.js";
export const UPSTREAM_TIMEOUT_MS = 12 * 60 * 1000;

const liveUpstreamTimers = new Set<ReturnType<typeof setTimeout>>();
Expand Down Expand Up @@ -127,6 +128,7 @@ export async function fetchWithTimeout(
timeoutMs?: number,
externalSignal?: AbortSignal,
): Promise<{ response: Response; clearTimer: () => void; stopIdleTimer: () => void }> {
checkOutboundBody(opts.body);
const effective = timeoutMs ?? upstreamTimeoutMs();
const controller = new AbortController();
let cleared = false;
Expand Down
39 changes: 39 additions & 0 deletions src/request-body-budget.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
export const MAX_REQUEST_BYTES = 100 * 1024 * 1024;
export const MAX_DECODED_REQUEST_BYTES = 2 * MAX_REQUEST_BYTES;
const MAX_LARGE_DECODED_REQUESTS = 2;
let largeDecodedRequests = 0;

export class DecodedRequestBusyError extends Error {
constructor() {
super("large decoded request budget is busy; retry after an active request finishes");
this.name = "DecodedRequestBusyError";
}
}

/** 只限制超过旧上限的解压请求;普通请求不占超大正文名额。 */
export class DecodedRequestAdmission {
private admitted = false;
observe(bytes: number): void {
if (bytes <= MAX_REQUEST_BYTES || this.admitted) return;
if (largeDecodedRequests >= MAX_LARGE_DECODED_REQUESTS) throw new DecodedRequestBusyError();
largeDecodedRequests++;
this.admitted = true;
}
release(): void {
if (!this.admitted) return;
largeDecodedRequests--;
this.admitted = false;
}
}

export class OutboundBodyTooLargeError extends Error {
constructor(public readonly bytes: number) {
super(`rebuilt request exceeds ${MAX_REQUEST_BYTES} bytes (${bytes} bytes); reduce historical images or content before retrying`);
this.name = "OutboundBodyTooLargeError";
}
}

export function checkOutboundBody(body: unknown): void {
const bytes = typeof body === "string" ? Buffer.byteLength(body) : Buffer.isBuffer(body) ? body.byteLength : 0;
if (bytes > MAX_REQUEST_BYTES) throw new OutboundBodyTooLargeError(bytes);
}
42 changes: 38 additions & 4 deletions src/server/handle.ts
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ import { FALLBACK_EFFECTIVE_WINDOW_FLOOR, findRoute, findRouteKey, lookupContext
import { resolveProxyDecision } from "../upstream-proxy.js";
import { contextFromRegistry, peekRegistryContext, peekRegistryOutputLimit, peekRegistryPriceProfile } from "../registry.js";
import { codexAlignedWindow } from "../codex-models.js";
import { MAX_REQUEST_BYTES } from "../fetch-util.js";
import { DecodedRequestAdmission, DecodedRequestBusyError, MAX_DECODED_REQUEST_BYTES } from "../request-body-budget.js";
import { hostIdForLog, maskHeadersForLog, maskUrlForLog, maskUrlsInText } from "../log-mask.js";
import { buildIncomingImageIndex, foldAnchoredCutoff, pruneRetrieveImgExports } from "../image-restore.js";
import { durableMessageGuards } from "../durable-message-guards.js";
Expand Down Expand Up @@ -76,6 +76,33 @@ export async function handle(
proxyWatchers: Set<number>,
initialWatcherPid: number | null,
adminCtx: Parameters<typeof handleAdminRoute>[2],
): Promise<void> {
const admission = new DecodedRequestAdmission();
const decodingAbort = new AbortController();
const close = () => { if (!res.writableEnded) decodingAbort.abort(); };
res.on("close", close);
try {
await handleRequest(req, res, opts, core, config, log, instanceId, instanceStartedAt, proxyWatchers, initialWatcherPid, adminCtx, admission, decodingAbort.signal);
} finally {
res.removeListener("close", close);
admission.release();
}
}

async function handleRequest(
req: http.IncomingMessage,
res: http.ServerResponse,
opts: ProxyOptions,
core: CompressionCore,
config: Config,
log: (level: string, msg: string) => void,
instanceId: string,
instanceStartedAt: number,
proxyWatchers: Set<number>,
initialWatcherPid: number | null,
adminCtx: Parameters<typeof handleAdminRoute>[2],
admission: DecodedRequestAdmission,
decodingSignal: AbortSignal,
): Promise<void> {
// #1440 P2 cut 1: the /__bili/* + /__acp/* management surface (security gates +
// endpoint dispatch) moved to src/server/admin.ts behind the loopback +
Expand Down Expand Up @@ -174,11 +201,11 @@ export async function handle(
// (e.g. GET /models) must forward raw bytes without content-encoding decode.
if (!passthroughMark && protocol !== null && bodyBuffer.length > 0) {
try {
const decoded = await decodeRequestBody(headerValue(req, "content-encoding"), bodyBuffer, MAX_REQUEST_BYTES);
const decoded = await decodeRequestBody(headerValue(req, "content-encoding"), bodyBuffer, MAX_DECODED_REQUEST_BYTES, { signal: decodingSignal, onBytes: (bytes) => admission.observe(bytes) });
bodyBuffer = decoded.body;
if (decoded.decoded) delete req.headers["content-encoding"];
} catch (decErr) {
if (decErr instanceof DecompressedTooLargeError) throw decErr;
if (decErr instanceof DecompressedTooLargeError || decErr instanceof DecodedRequestBusyError || decodingSignal.aborted) throw decErr;
// #619: bili can't decode this content-encoding -> don't 400. Drop
// protocol so the request falls to the verbatim passthrough below,
// relaying the ORIGINAL still-encoded bytes (the reassignment never
Expand All @@ -189,10 +216,17 @@ export async function handle(
}
}
} catch (err) {
if (decodingSignal.aborted || res.destroyed) return;
if (err instanceof DecodedRequestBusyError) {
log("warn", "503: large decoded request budget is busy");
res.writeHead(503, { "content-type": "application/json", "retry-after": "1" });
res.end(JSON.stringify({ error: { type: "request_body_busy", stage: "decode", message: err.message } }));
return;
}
if (err instanceof BodyTooLargeError || err instanceof DecompressedTooLargeError) {
log("warn", `413: request body exceeds ${err.limit} bytes`);
res.writeHead(413, { "content-type": "application/json" });
res.end(JSON.stringify({ error: { type: "request_too_large", message: err.message } }));
res.end(JSON.stringify({ error: { type: "request_too_large", stage: err instanceof BodyTooLargeError ? "receive" : "decode", message: err.message } }));
return;
}
log("warn", `failed to prepare inbound request (${String(err)}) - 400`);
Expand Down
14 changes: 14 additions & 0 deletions src/server/relay.ts
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
import { checkOutboundBody, OutboundBodyTooLargeError } from "../request-body-budget.js";
// #1440 P2 cut 4: upstream relay zone (forward) extracted verbatim from src/server.ts — pure move, zero behavior change.

import http from "node:http";
Expand Down Expand Up @@ -267,6 +268,7 @@ export async function forward(
log("debug", `[${prepared.session.id}] [chain] outbound stamping failed (${String(err)}); forwarding unstamped`);
}
}
if (res.destroyed || res.writableEnded) return;
// #552: wire transform shared by ALL re-send paths (compress-retry loops
// below) so re-sent bodies carry the same rewrite as the initial forward —
// otherwise a developer-role 400 would hit mid-stream on the first retry.
Expand Down Expand Up @@ -392,6 +394,18 @@ export async function forward(
if (typeof wireBody === "string") wireBody = scrubLoneSurrogatesOnWire(wireBody);
applyResign(headers, wireBody);
}
try {
checkOutboundBody(wireBody);
} catch (err) {
if (!(err instanceof OutboundBodyTooLargeError)) throw err;
log("warn", `413: ${err.message}`);
if (res.headersSent) emitStreamError(res, prepared?.protocol ?? "openai", err.message, undefined, opts.streamErrorShape);
else {
res.writeHead(413, { "content-type": "application/json" });
res.end(JSON.stringify({ error: { type: "request_too_large", stage: "forward", message: err.message } }));
}
return;
}
const init: Omit<RequestInit, "dispatcher"> & { dispatcher?: object } = {
method: req.method ?? "GET",
headers,
Expand Down
Loading
Loading