diff --git a/AGENTS.md b/AGENTS.md index 46beba4e0c..5bdabee1cf 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -222,6 +222,7 @@ Commit convention: `feat:` / `fix:` / `refactor:` / `test:` / `docs:` / `release ### 7.3 Correctness Guardrails +- **Availability over performance (可用性永远大于性能 — owner rule, 2026-10-14, #3128).** When a defect makes a capability UNAVAILABLE, restoring availability outranks every other consideration — performance, memory footprint, elegance. Do not block or defer an availability fix on performance grounds: ship the availability fix and track the performance cost as SEPARATE work (it may already be in flight). Canonical case #3128: raising the hardcoded 100 MiB HTTP request-body cap to a 5 GiB default so multi-gigabyte single requests are accepted instead of reset is an AVAILABILITY decision; the in-RAM buffering cost of large single bodies is a distinct performance concern owned by separate streaming/spill work, not a reason to hold the cap. - **Root cause, not symptom (疏, not 堵 — first principle for every fix).** A fix removes the mechanism that PRODUCES the defect; it never merely hides the defect's visible trace at whatever layer is cheapest to intercept. Before writing any fix, name the producer of the offending bytes/behavior and fix it at the source or decision point — never by rewriting or filtering payload in transit to make the symptom invisible. Litmus test: "if the same root cause surfaced in a different shape tomorrow, would this fix still hold?" If not, it is a mask, and a mask is a latent bug. Canonical failure: #933 (model-echoed render tags leaking into a TUI) was "fixed" by stripping tool-call arguments mid-stream; the mask itself became #1039's silent data corruption once users legitimately wrote tag-shaped payloads through write/edit/bash. When the true root cause is out of scope (another repo, a product decision), fix the in-scope part, leave a documented boundary note pointing at the real home of the fix, and do NOT compensate in the wrong layer (#1039's echo-noise belongs on the injection side — host renderTags policy — not on the wire). - **Never silently clobber or drop user config.** Any read-modify-write on user config needs a parse-state guard; reject malformed input loudly (HTTP 400/409) instead of merging into defaults or dropping fields. Whitelists must be complete (#155: a missing key silently erased custom compress prompts). - **Sane defaults & fallbacks.** Fallback values must be reasonable (never a too-small value that thrashes); a static/fallback source must always lose to a fresher authoritative source when both are cheaply available (#282: window fallback 200K/min 100K, not 64K; the bundled registry snapshot must not outrank the live registry). diff --git a/CONFIGURATION.md b/CONFIGURATION.md index f0d3b3440c..c7b3376dca 100644 --- a/CONFIGURATION.md +++ b/CONFIGURATION.md @@ -153,6 +153,9 @@ This index is generated from `website/config-reference/*.yaml` — edit the seed | `network.proxyKeepAliveMaxMs` | number | 55000 (0 = one-shot connections) | BILI_PROXY_KEEPALIVE_MAX_MS | Max lifetime of a connection through an outbound egress proxy before forcing reconnection (avoids stale NAT mappings). | | `network.postResponseLingerMs` | number | 5000 | BILI_POST_RESPONSE_LINGER_MS | Graceful-close budget for proxy-initiated closes after the response completed: the socket is held waiting for the peer's FIN/TLS close_notify and destroyed at the deadline (reason=linger-backstop). | | `network.allowPrivateHttpProviders` | boolean | false | BILI_ALLOW_PRIVATE_HTTP_PROVIDERS | Opt-in (#2984): admit plain-HTTP providers on private networks (RFC1918, 127/8, link-local, ULA, *.local/.lan/.internal) into the agent-registry intake and summary-endpoint guards. Off by default — plaintext stays loopback-only; public IPs/domains and CGNAT (100.64/10) stay refused regardless. 1/true enables. | +| `network.receiveBodyBytes` | number | 5368709120 | BILI_RECEIVE_BODY_BYTES | Max inbound wire bytes read into memory per request before a 413 at the receive stage. Raised from a hardcoded 100 MiB to 5 GiB (#3128 owner decision) so multi-gigabyte single requests are accepted instead of the socket being reset. Also the threshold above which a decoded request consumes one of two shared large-decode admission slots. WebSocket lanes keep the fixed 100 MiB surface. | +| `network.decodedBodyBytes` | number | 10737418240 | BILI_DECODED_BODY_BYTES | Max DECODED bytes after content-encoding expansion (gzip/deflate/br/zstd), checked incrementally during decode (decode-stage 413). Default is 2x the receive budget, mirroring the legacy decode=2x-wire relationship; guards against decompression bombs. | +| `network.outboundBodyBytes` | number | 5368709120 | BILI_OUTBOUND_BODY_BYTES | Max rebuilt request bytes forwarded upstream (forward-stage 413), counted as UTF-8 bytes of the final body string. Checked before the fetch so an over-budget rebuild never reaches the upstream watchdog allocation. | | `persist` | object | {} (all defaults) | — | Session persistence to disk under the XDG state dir. | | `persist.enabled` | boolean | true | BILI_PERSIST | Master switch for writing session records to disk (0/false turns it off). | | `persist.zstd` | boolean | false | BILI_PERSIST_ZSTD | Zstd-compress persisted session files (1/true enables). | @@ -1642,6 +1645,7 @@ File keys resolve only when the matching env var is unset. Defaults in parenthes | `BILI_CLIENT_ERROR_BACKSTOP_MS` | `network.clientErrorBackstopMs` | 30000 | | `BILI_CODEARTS_REF` | `resign` | {} (armed; built-in scheme sdk-hmac-sha256) | | `BILI_CODEX_COMPACT` | `codexCompact` | "intercept" | +| `BILI_DECODED_BODY_BYTES` | `network.decodedBodyBytes` | 10737418240 | | `BILI_DECOMPRESS_TMP_CAP` | `decompressTmpCap` | 50 | | `BILI_DUMP_4XX` | `diagnostics.dump4xx` | false | | `BILI_DUMP_4XX_MAX_BYTES` | `diagnostics.dump4xxMaxBytes` | 2097152 (floor 1024) | @@ -1666,6 +1670,7 @@ File keys resolve only when the matching env var is unset. Defaults in parenthes | `BILI_NATIVE_ATTACH_POLICY` | `native.attachPolicy` | own-lane | | `BILI_NON_HTTP_PROVIDERS` | `compactionOptIn` | false | | `BILI_NO_CACHE_CONTROL` | `compat.noCacheControl` | false | +| `BILI_OUTBOUND_BODY_BYTES` | `network.outboundBodyBytes` | 5368709120 | | `BILI_PERSIST` | `persist.enabled` | true | | `BILI_PERSIST_DEBOUNCE_MS` | `persist.debounceMs` | 500 | | `BILI_PERSIST_EPERM_ALERT_REPEAT_MS` | `persist.epermAlertRepeatMs` | 0 (no repeats) | @@ -1678,6 +1683,7 @@ File keys resolve only when the matching env var is unset. Defaults in parenthes | `BILI_PREFLIGHT_HOLD_MS` | `network.preflightHoldMs` | 30000 | | `BILI_PROXY_KEEPALIVE_MAX_MS` | `network.proxyKeepAliveMaxMs` | 55000 (0 = one-shot connections) | | `BILI_PUBLIC_SNAPSHOT_CAP_BYTES` | `plugin.snapshotCapBytes` | 104857600 (0 disables snapshots) | +| `BILI_RECEIVE_BODY_BYTES` | `network.receiveBodyBytes` | 5368709120 | | `BILI_RELEASE_NOTES_CHECK` | `releaseNotesCheck` | true | | `BILI_RELEASE_NOTES_URL` | `releaseNotesUrl` | unset (built-in feed) | | `BILI_REPLAY_RETRY_BASE_MS` | `network.replayRetryBaseMs` | 1500 (0 disables the delay) | diff --git a/CONFIGURATION.zh-CN.md b/CONFIGURATION.zh-CN.md index 475f7d23f1..2513b1a773 100644 --- a/CONFIGURATION.zh-CN.md +++ b/CONFIGURATION.zh-CN.md @@ -153,6 +153,9 @@ | `network.proxyKeepAliveMaxMs` | number | 55000 (0 = one-shot connections) | BILI_PROXY_KEEPALIVE_MAX_MS | 经出站代理的连接最大存活时长,到期强制重建(规避陈旧 NAT 映射)。 | | `network.postResponseLingerMs` | number | 5000 | BILI_POST_RESPONSE_LINGER_MS | 代理主动发起的响应后关闭的优雅关闭预算:socket 挂起等待对端 FIN/TLS close_notify,到点销毁(reason=linger-backstop)。 | | `network.allowPrivateHttpProviders` | boolean | false | BILI_ALLOW_PRIVATE_HTTP_PROVIDERS | 可选开关(#2984):允许私网(RFC1918、127/8、link-local、ULA、*.local/.lan/.internal)明文 HTTP provider 进入 agent-registry 接收与摘要端点守卫。默认关——明文仅回环受信;公网 IP/域名与 CGNAT(100.64/10)无论如何都拒。1/true 启用。 | +| `network.receiveBodyBytes` | number | 5368709120 | BILI_RECEIVE_BODY_BYTES | 单请求读入内存的入站 wire 字节上限,超限在 receive 阶段返回 413。由硬编码 100 MiB 提升到 5 GiB(#3128 owner 决定),使多 GB 单请求被接受而非断连。同时也是解码请求占用两个共享大解码准入槽位的阈值。WebSocket lane 保持固定 100 MiB 表面。 | +| `network.decodedBodyBytes` | number | 10737418240 | BILI_DECODED_BODY_BYTES | content-encoding 解压(gzip/deflate/br/zstd)后的解码字节上限,解码过程中增量检查(decode 阶段 413)。默认为接收预算的 2 倍,沿用旧的 decode=2×wire 关系;用于防解压炸弹。 | +| `network.outboundBodyBytes` | number | 5368709120 | BILI_OUTBOUND_BODY_BYTES | 转发到上游的重建请求字节上限(forward 阶段 413),按最终 body 字符串的 UTF-8 字节计。在 fetch 之前检查,超预算的重建不会触发上游看门狗分配。 | | `persist` | object | {} (all defaults) | — | 会话落盘持久化(XDG state 目录之下)。 | | `persist.enabled` | boolean | true | BILI_PERSIST | 会话记录落盘总开关(0/false 关闭)。 | | `persist.zstd` | boolean | false | BILI_PERSIST_ZSTD | zstd 压缩持久化会话文件(1/true 启用)。 | @@ -1641,6 +1644,7 @@ ACP 原生 agent(当前为 `pi` 扩展)会在每个进程内向代理上报 | `BILI_CLIENT_ERROR_BACKSTOP_MS` | `network.clientErrorBackstopMs` | 30000 | | `BILI_CODEARTS_REF` | `resign` | {} (armed; built-in scheme sdk-hmac-sha256) | | `BILI_CODEX_COMPACT` | `codexCompact` | "intercept" | +| `BILI_DECODED_BODY_BYTES` | `network.decodedBodyBytes` | 10737418240 | | `BILI_DECOMPRESS_TMP_CAP` | `decompressTmpCap` | 50 | | `BILI_DUMP_4XX` | `diagnostics.dump4xx` | false | | `BILI_DUMP_4XX_MAX_BYTES` | `diagnostics.dump4xxMaxBytes` | 2097152 (floor 1024) | @@ -1665,6 +1669,7 @@ ACP 原生 agent(当前为 `pi` 扩展)会在每个进程内向代理上报 | `BILI_NATIVE_ATTACH_POLICY` | `native.attachPolicy` | own-lane | | `BILI_NON_HTTP_PROVIDERS` | `compactionOptIn` | false | | `BILI_NO_CACHE_CONTROL` | `compat.noCacheControl` | false | +| `BILI_OUTBOUND_BODY_BYTES` | `network.outboundBodyBytes` | 5368709120 | | `BILI_PERSIST` | `persist.enabled` | true | | `BILI_PERSIST_DEBOUNCE_MS` | `persist.debounceMs` | 500 | | `BILI_PERSIST_EPERM_ALERT_REPEAT_MS` | `persist.epermAlertRepeatMs` | 0 (no repeats) | @@ -1677,6 +1682,7 @@ ACP 原生 agent(当前为 `pi` 扩展)会在每个进程内向代理上报 | `BILI_PREFLIGHT_HOLD_MS` | `network.preflightHoldMs` | 30000 | | `BILI_PROXY_KEEPALIVE_MAX_MS` | `network.proxyKeepAliveMaxMs` | 55000 (0 = one-shot connections) | | `BILI_PUBLIC_SNAPSHOT_CAP_BYTES` | `plugin.snapshotCapBytes` | 104857600 (0 disables snapshots) | +| `BILI_RECEIVE_BODY_BYTES` | `network.receiveBodyBytes` | 5368709120 | | `BILI_RELEASE_NOTES_CHECK` | `releaseNotesCheck` | true | | `BILI_RELEASE_NOTES_URL` | `releaseNotesUrl` | unset (built-in feed) | | `BILI_REPLAY_RETRY_BASE_MS` | `network.replayRetryBaseMs` | 1500 (0 disables the delay) | diff --git a/src/config.ts b/src/config.ts index e46b4b1af1..4c45ee643b 100644 --- a/src/config.ts +++ b/src/config.ts @@ -1829,6 +1829,14 @@ type FileConfig = { * guards. Default false: plaintext stays loopback-only. Public IPs/domains and * CGNAT (100.64/10) remain refused regardless. Env BILI_ALLOW_PRIVATE_HTTP_PROVIDERS wins. */ allowPrivateHttpProviders?: boolean; + /** Per-request inbound wire byte budget (#3128; was hardcoded 100 MiB, now default 5 GiB). + * Env BILI_RECEIVE_BODY_BYTES wins. Also the threshold above which a decoded request + * consumes one of two shared large-decode admission slots. WebSocket lanes keep 100 MiB. */ + receiveBodyBytes?: number; + /** Per-request DECODED byte budget after content-encoding expansion (#3128; default 10 GiB = 2x receive). Env BILI_DECODED_BODY_BYTES wins. */ + decodedBodyBytes?: number; + /** Per-request rebuilt-outbound byte budget forwarded upstream (#3128; default 5 GiB). Env BILI_OUTBOUND_BODY_BYTES wins. */ + outboundBodyBytes?: number; }; /** Session persistence knobs (#2030) — was BILI_PERSIST_* env-only. */ persist?: { diff --git a/src/knobs.ts b/src/knobs.ts index 11982a08f4..0ff62c4a55 100644 --- a/src/knobs.ts +++ b/src/knobs.ts @@ -170,6 +170,36 @@ export function mitmHandshakeTimeoutMs(): number { return Number.isFinite(v) && v > 0 ? Math.floor(v) : MITM_HANDSHAKE_TIMEOUT_MS_DEFAULT; } +// --- request-body byte budgets (#3128) --- + +const DEFAULT_LIMITS_RECEIVE_BYTES = 5 * 1024 * 1024 * 1024; +const DEFAULT_LIMITS_DECODED_BYTES = 10 * 1024 * 1024 * 1024; +const DEFAULT_LIMITS_OUTBOUND_BYTES = 5 * 1024 * 1024 * 1024; + +/** BILI_RECEIVE_BODY_BYTES > network.receiveBodyBytes > 5 GiB. Max inbound wire bytes + * read into memory per request. #3128: these were hardcoded 100 MiB constants; the + * HTTP path now resolves them here with a 5 GiB default so multi-gigabyte single + * requests are accepted instead of reset. They bound SINGLE-REQUEST in-RAM buffering + * (~2x peak while a body is read+decoded); cross-turn history lives in SQLite and is + * unaffected. The WebSocket lanes keep the fixed 100 MiB surface (MAX_REQUEST_BYTES). */ +export function limitsReceiveBytes(): number { + const v = tInt("BILI_RECEIVE_BODY_BYTES", loadConfigFile().network?.receiveBodyBytes); + return Number.isFinite(v) && v > 0 ? Math.floor(v) : DEFAULT_LIMITS_RECEIVE_BYTES; +} + +/** BILI_DECODED_BODY_BYTES > network.decodedBodyBytes > 10 GiB. Max DECODED bytes after + * content-encoding expansion (2x the receive default, mirroring the legacy decode cap). */ +export function limitsDecodedBytes(): number { + const v = tInt("BILI_DECODED_BODY_BYTES", loadConfigFile().network?.decodedBodyBytes); + return Number.isFinite(v) && v > 0 ? Math.floor(v) : DEFAULT_LIMITS_DECODED_BYTES; +} + +/** BILI_OUTBOUND_BODY_BYTES > network.outboundBodyBytes > 5 GiB. Max rebuilt bytes forwarded upstream. */ +export function limitsOutboundBytes(): number { + const v = tInt("BILI_OUTBOUND_BODY_BYTES", loadConfigFile().network?.outboundBodyBytes); + return Number.isFinite(v) && v > 0 ? Math.floor(v) : DEFAULT_LIMITS_OUTBOUND_BYTES; +} + // --- session persistence (was BILI_PERSIST_* env-only) --- function filePersist() { diff --git a/src/request-body-budget.ts b/src/request-body-budget.ts index acad1dcb46..b51d6d1a20 100644 --- a/src/request-body-budget.ts +++ b/src/request-body-budget.ts @@ -1,3 +1,5 @@ +import { limitsOutboundBytes, limitsReceiveBytes } from "./knobs.js"; + export const MAX_REQUEST_BYTES = 100 * 1024 * 1024; export const MAX_DECODED_REQUEST_BYTES = 2 * MAX_REQUEST_BYTES; const MAX_LARGE_DECODED_REQUESTS = 2; @@ -10,11 +12,11 @@ export class DecodedRequestBusyError extends Error { } } -/** 只限制超过旧上限的解压请求;普通请求不占超大正文名额。 */ +/** 只限制超过配置的接收预算(limitsReceiveBytes)的解压请求;普通请求不占超大正文名额。 */ export class DecodedRequestAdmission { private admitted = false; observe(bytes: number): void { - if (bytes <= MAX_REQUEST_BYTES || this.admitted) return; + if (bytes <= limitsReceiveBytes() || this.admitted) return; if (largeDecodedRequests >= MAX_LARGE_DECODED_REQUESTS) throw new DecodedRequestBusyError(); largeDecodedRequests++; this.admitted = true; @@ -28,12 +30,12 @@ export class DecodedRequestAdmission { export class OutboundBodyTooLargeError extends Error { constructor(public readonly bytes: number) { - super(`rebuilt request exceeds ${MAX_REQUEST_BYTES} bytes (${bytes} bytes); reduce historical images or content before retrying`); + super(`rebuilt request exceeds ${limitsOutboundBytes()} bytes (${bytes} bytes); reduce historical images or content before retrying`); this.name = "OutboundBodyTooLargeError"; } } export function checkOutboundBody(body: unknown): void { const bytes = typeof body === "string" ? Buffer.byteLength(body) : Buffer.isBuffer(body) ? body.byteLength : 0; - if (bytes > MAX_REQUEST_BYTES) throw new OutboundBodyTooLargeError(bytes); + if (bytes > limitsOutboundBytes()) throw new OutboundBodyTooLargeError(bytes); } diff --git a/src/server.ts b/src/server.ts index c91002d970..0a3a51f9b5 100644 --- a/src/server.ts +++ b/src/server.ts @@ -15,7 +15,7 @@ import { resetProxyCache } from "./upstream-proxy.js"; import { FALLBACK_EFFECTIVE_WINDOW_FLOOR, findRoute, findRouteKey, lookupContextLimit, resolveConfiguredContextLimit, resolveConfiguredOutputLimit, resolveCompressProtocol, resolveResignSettings } from "./config.js"; import { contextFromRegistry, loadRegistry, peekRegistryContext, peekRegistryOutputLimit, peekRegistryPriceProfile } from "./registry.js"; import { codexAlignedWindow } from "./codex-models.js"; -import { fetchWithTimeout, MAX_REQUEST_BYTES, upstreamTimeoutMs } from "./fetch-util.js"; +import { fetchWithTimeout, upstreamTimeoutMs } from "./fetch-util.js"; import { getUpstreamConnectionStatus, resolveProxy, resolveProxyDecision, proxyDispatcher, type UpstreamProxyDecision } from "./upstream-proxy.js"; import type { SkippedProxyInfo } from "./upstream-fail.js"; import { getUpstreamAlerts } from "./upstream-alerts.js"; @@ -134,6 +134,7 @@ import { forceTextProtocol as knobForceTextProtocol, keepAliveTimeoutMs as knobKeepAliveTimeoutMs, keepResponseId as knobKeepResponseId, + limitsReceiveBytes as knobLimitsReceiveBytes, noCompressPrompt as knobNoCompressPrompt, noInjectTool as knobNoInjectTool, postResponseLingerMs as knobPostResponseLingerMs, @@ -3521,10 +3522,10 @@ export function logRequestCost(log: (level: string, msg: string) => void, sessio log("info", `[${sessionId}] request: ${msgs ?? "?"} msgs, inbound=${formatBytes(inboundBytes)}${outboundField}, local=${ms}ms${view}`); } -/** Thrown by readBody when the request body exceeds MAX_REQUEST_BYTES. - * handle() catches this and attempts a 413 response; readBody also destroys - * the request socket so a client that keeps streaming a pathological body - * cannot hold the connection open. */ +/** Thrown by readBody when the request body exceeds the configured receive-body + * budget (limitsReceiveBytes). handle() catches this and attempts a 413 response; + * readBody also destroys the request socket so a client that keeps streaming a + * pathological body cannot hold the connection open. */ export class BodyTooLargeError extends Error { constructor(public readonly limit: number) { super(`request body exceeds ${limit} bytes`); @@ -3540,10 +3541,10 @@ export function readBody(req: http.IncomingMessage): Promise { req.on("data", (c: Buffer) => { if (aborted) return; size += c.length; - if (size > MAX_REQUEST_BYTES) { + if (size > knobLimitsReceiveBytes()) { aborted = true; req.destroy(); - reject(new BodyTooLargeError(MAX_REQUEST_BYTES)); + reject(new BodyTooLargeError(knobLimitsReceiveBytes())); return; } chunks.push(c); diff --git a/src/server/handle.ts b/src/server/handle.ts index eb5ae27eff..876f7ddd3c 100644 --- a/src/server/handle.ts +++ b/src/server/handle.ts @@ -12,7 +12,7 @@ import { FALLBACK_EFFECTIVE_WINDOW_FLOOR, findRoute, findRouteKey, lookupContext import { resolveProxyDecision } from "../upstream-proxy.js"; import { contextFromRegistry, peekRegistryContext, peekRegistryOutputLimit, peekRegistryPriceProfile } from "../registry.js"; import { codexAlignedWindow } from "../codex-models.js"; -import { DecodedRequestAdmission, DecodedRequestBusyError, MAX_DECODED_REQUEST_BYTES } from "../request-body-budget.js"; +import { DecodedRequestAdmission, DecodedRequestBusyError } from "../request-body-budget.js"; import { hostIdForLog, maskHeadersForLog, maskUrlForLog, maskUrlsInText } from "../log-mask.js"; import { bodyMightCarryImages, buildIncomingImageIndex, pruneRetrieveImgExports, refreshIncomingImageIndex, type IndexedImage } from "../image-restore.js"; import { durableMessageToolPaths, withDurableProtection } from "../durable-message-guards.js"; @@ -45,7 +45,7 @@ import { BILI_PASSTHROUGH_HEADER, BILI_PLUGIN_BYPASS_HEADER, isLoopbackAddress, import { checkTunnelDestination, tunnelAllowlistFromEnv } from "../tunnel-guard.js"; import { resolveCompatDropFields } from "../compat-drop.js"; import { DecompressedTooLargeError, decodeRequestBody } from "../content-encoding.js"; -import { autoModeForgeEnabled, noInjectTool as knobNoInjectTool, rawDumpDir as knobRawDumpDir } from "../knobs.js"; +import { autoModeForgeEnabled, limitsDecodedBytes as knobLimitsDecodedBytes, noInjectTool as knobNoInjectTool, rawDumpDir as knobRawDumpDir } from "../knobs.js"; import { buildForgeResponse, classifierForgeVariant, estimateCjkTokens, isAutoModeClassifier, requestText, summarizeAction } from "../auto-mode-forge.js"; import { extractConversationId, isV1BackgroundSideRequest, isV1InternalRequest, unwrapV1Internal } from "../v1internal.js"; import { anthropicBetaContextWindow, BILI_HOP_HEADER, capRegistryWindowByStandard, expandedContextSuffixWindow, launcherContextWindow, launcherMaxOutput, windowSourceLogged } from "./context-window.js"; @@ -312,7 +312,7 @@ async function handleRequest( // (e.g. GET /models) must forward raw bytes without content-encoding decode. if (!passthroughMark && protocol !== null && bodyBuffer.length > 0) { try { - const decoded = await decodeRequestBody(headerValue(req, "content-encoding"), bodyBuffer, MAX_DECODED_REQUEST_BYTES, { signal: decodingSignal, onBytes: (bytes) => admission.observe(bytes) }); + const decoded = await decodeRequestBody(headerValue(req, "content-encoding"), bodyBuffer, knobLimitsDecodedBytes(), { signal: decodingSignal, onBytes: (bytes) => admission.observe(bytes) }); bodyBuffer = decoded.body; if (decoded.decoded) delete req.headers["content-encoding"]; } catch (decErr) { @@ -366,7 +366,7 @@ async function handleRequest( // relay verbatim, today's behavior). The check decodes only a COPY: the // relay still forwards the original bytes with content-encoding intact. if (protocol !== null && req.method === "POST" && bodyBuffer.length > 0 && opts.allowDshCompaction !== true - && await detectPassthroughDshCompactionCall(protocol, bodyBuffer, headerValue(req, "content-encoding"), async (enc, buf) => (await decodeRequestBody(enc, buf, MAX_DECODED_REQUEST_BYTES, { signal: decodingSignal, onBytes: (bytes) => admission.observe(bytes) })).body)) { + && await detectPassthroughDshCompactionCall(protocol, bodyBuffer, headerValue(req, "content-encoding"), async (enc, buf) => (await decodeRequestBody(enc, buf, knobLimitsDecodedBytes(), { signal: decodingSignal, onBytes: (bytes) => admission.observe(bytes) })).body)) { dshPassthroughGuardStats.refusals += 1; log("warn", `[passthrough] dsh native compaction call identified on the unattributed passthrough lane (#1117/#2709) — REFUSED, not forwarded: bili owns compression on this lane; a landed dsh checkpoint would durably shadow the raw history (#1729, cf. #1772/#1206). No session exists on this lane, so this log line (not the #2490 ledger) is the record; passthrough refusals so far: ${dshPassthroughGuardStats.refusals}`); const refusal = dshCompactionRefusal(protocol); diff --git a/tests/decode-fail-passthrough.test.ts b/tests/decode-fail-passthrough.test.ts index 6e61e07197..a319a8c4d9 100644 --- a/tests/decode-fail-passthrough.test.ts +++ b/tests/decode-fail-passthrough.test.ts @@ -8,6 +8,13 @@ import { SessionStore, _setStoreForTest } from "../src/persist.ts"; import type { ProxyOptions } from "../src/config.ts"; import { _setForTest as setRegistryForTest } from "../src/registry.ts"; +// #3128: the oversized-decompressed-body 413 below is asserted at the LEGACY caps. Pin the +// knobs to the old 100/200/100 MiB values so this stays independent of the new 5 GiB shipped +// defaults (covered by knobs-config-precedence); small bodies in the other tests are unaffected. +process.env.BILI_RECEIVE_BODY_BYTES = "104857600"; +process.env.BILI_DECODED_BODY_BYTES = "209715200"; +process.env.BILI_OUTBOUND_BODY_BYTES = "104857600"; + type Captured = { url: string; headers: http.IncomingHttpHeaders; body: Buffer }; function listen(server: http.Server): Promise { diff --git a/tests/image-request-budget.test.ts b/tests/image-request-budget.test.ts index 4165ffe72d..64cc64de94 100644 --- a/tests/image-request-budget.test.ts +++ b/tests/image-request-budget.test.ts @@ -14,6 +14,13 @@ import type { ProxyOptions } from "../src/config.ts"; import { _setForTest as setRegistryForTest } from "../src/registry.ts"; import { DecodedRequestAdmission, MAX_REQUEST_BYTES } from "../src/request-body-budget.ts"; +// #3128: assert the LEGACY byte-budget behavior at the old 100/200/100 MiB caps (receive / +// decoded / outbound). Pin the knobs to those exact values so every stage below stays +// independent of the new 5 GiB shipped defaults (covered by knobs-config-precedence). +process.env.BILI_RECEIVE_BODY_BYTES = "104857600"; +process.env.BILI_DECODED_BODY_BYTES = "209715200"; +process.env.BILI_OUTBOUND_BODY_BYTES = "104857600"; + const isolation = mkdtempSync(join(tmpdir(), "bili-request-budget-")); process.env.XDG_STATE_HOME = join(isolation, "state"); process.env.XDG_DATA_HOME = join(isolation, "data"); diff --git a/tests/knobs-config-precedence.test.ts b/tests/knobs-config-precedence.test.ts index b0a359fb8d..c3463e7f26 100644 --- a/tests/knobs-config-precedence.test.ts +++ b/tests/knobs-config-precedence.test.ts @@ -41,6 +41,34 @@ const knobs = await import("../src/knobs.js"); const { UPSTREAM_TIMEOUT_MS, REPLAY_MAX_ATTEMPTS, REPLAY_BASE_DELAY_MS } = await import("../src/fetch-util.js"); const { PROXY_KEEPALIVE_MAX_MS } = await import("../src/upstream-proxy.js"); +test("#3128 limits: defaults 5/10/5 GiB, file tier, env precedence, garbage falls back", () => { + clearConfig(); + assert.equal(knobs.limitsReceiveBytes(), 5 * 1024 * 1024 * 1024); + assert.equal(knobs.limitsDecodedBytes(), 10 * 1024 * 1024 * 1024); + assert.equal(knobs.limitsOutboundBytes(), 5 * 1024 * 1024 * 1024); + + setConfig({ network: { receiveBodyBytes: 111, decodedBodyBytes: 222, outboundBodyBytes: 333 } }); + assert.equal(knobs.limitsReceiveBytes(), 111); + assert.equal(knobs.limitsDecodedBytes(), 222); + assert.equal(knobs.limitsOutboundBytes(), 333); + + withEnv({ BILI_RECEIVE_BODY_BYTES: "999", BILI_DECODED_BODY_BYTES: "888", BILI_OUTBOUND_BODY_BYTES: "777" }, () => { + assert.equal(knobs.limitsReceiveBytes(), 999); + assert.equal(knobs.limitsDecodedBytes(), 888); + assert.equal(knobs.limitsOutboundBytes(), 777); + }); + + // non-numeric / non-positive values resolve to the default and never throw + setConfig({ network: { receiveBodyBytes: "big", decodedBodyBytes: -3, outboundBodyBytes: 0 } }); + assert.equal(knobs.limitsReceiveBytes(), 5 * 1024 * 1024 * 1024); + assert.equal(knobs.limitsDecodedBytes(), 10 * 1024 * 1024 * 1024); + assert.equal(knobs.limitsOutboundBytes(), 5 * 1024 * 1024 * 1024); + withEnv({ BILI_RECEIVE_BODY_BYTES: "junk" }, () => { + assert.equal(knobs.limitsReceiveBytes(), 5 * 1024 * 1024 * 1024); + }); + clearConfig(); +}); + test("defaults: no env, no file", () => { clearConfig(); assert.equal(knobs.upstreamTimeoutMs(), UPSTREAM_TIMEOUT_MS); diff --git a/tests/proxy-readbody.test.ts b/tests/proxy-readbody.test.ts index d674a8166b..2349ea2b55 100644 --- a/tests/proxy-readbody.test.ts +++ b/tests/proxy-readbody.test.ts @@ -5,6 +5,13 @@ import { EventEmitter } from "node:events"; import { BodyTooLargeError, readBody } from "../src/server.ts"; import { MAX_REQUEST_BYTES } from "../src/fetch-util.ts"; +// #3128: assert the LEGACY byte-budget behavior at the old 100 MiB cap. Pin the knob to +// that exact value so this stays independent of the new 5 GiB shipped default (the default +// itself is covered by knobs-config-precedence). node --test isolates each file's process. +process.env.BILI_RECEIVE_BODY_BYTES = "104857600"; +process.env.BILI_DECODED_BODY_BYTES = "209715200"; +process.env.BILI_OUTBOUND_BODY_BYTES = "104857600"; + test("readBody: returns buffer for a normal request", async () => { const buf = Buffer.from("hello"); const req = Readable.from([buf]) as never; diff --git a/tests/request-body-budget.test.ts b/tests/request-body-budget.test.ts index 78f115faab..9c8d680b48 100644 --- a/tests/request-body-budget.test.ts +++ b/tests/request-body-budget.test.ts @@ -5,6 +5,14 @@ import { decodeRequestBody, DecompressedTooLargeError } from "../src/content-enc import { checkOutboundBody, DecodedRequestAdmission, DecodedRequestBusyError, MAX_REQUEST_BYTES, MAX_DECODED_REQUEST_BYTES, OutboundBodyTooLargeError } from "../src/request-body-budget.ts"; import { fetchWithTimeout, _liveUpstreamTimersForTest } from "../src/fetch-util.ts"; +// #3128: assert the LEGACY byte-budget behavior at the old 100/200/100 MiB caps. Pin the +// knobs to those exact values so these stay independent of the new 5 GiB shipped defaults +// (the defaults themselves are covered by knobs-config-precedence). node --test isolates +// each file's process, so this cannot leak into other suites. +process.env.BILI_RECEIVE_BODY_BYTES = "104857600"; +process.env.BILI_DECODED_BODY_BYTES = "209715200"; +process.env.BILI_OUTBOUND_BODY_BYTES = "104857600"; + test("large decoded admission keeps normal requests free, limits two large requests, and releases idempotently", () => { const a = new DecodedRequestAdmission(), b = new DecodedRequestAdmission(), c = new DecodedRequestAdmission(); try { diff --git a/website/config-reference/process.yaml b/website/config-reference/process.yaml index 06ac46f27c..b25df247b9 100644 --- a/website/config-reference/process.yaml +++ b/website/config-reference/process.yaml @@ -110,6 +110,27 @@ entries: group: process desc_en: "Opt-in (#2984): admit plain-HTTP providers on private networks (RFC1918, 127/8, link-local, ULA, *.local/.lan/.internal) into the agent-registry intake and summary-endpoint guards. Off by default — plaintext stays loopback-only; public IPs/domains and CGNAT (100.64/10) stay refused regardless. 1/true enables." desc_zh: "可选开关(#2984):允许私网(RFC1918、127/8、link-local、ULA、*.local/.lan/.internal)明文 HTTP provider 进入 agent-registry 接收与摘要端点守卫。默认关——明文仅回环受信;公网 IP/域名与 CGNAT(100.64/10)无论如何都拒。1/true 启用。" + - key: network.receiveBodyBytes + type: number + default: "5368709120" + env: BILI_RECEIVE_BODY_BYTES + group: process + desc_en: "Max inbound wire bytes read into memory per request before a 413 at the receive stage. Raised from a hardcoded 100 MiB to 5 GiB (#3128 owner decision) so multi-gigabyte single requests are accepted instead of the socket being reset. Also the threshold above which a decoded request consumes one of two shared large-decode admission slots. WebSocket lanes keep the fixed 100 MiB surface." + desc_zh: "单请求读入内存的入站 wire 字节上限,超限在 receive 阶段返回 413。由硬编码 100 MiB 提升到 5 GiB(#3128 owner 决定),使多 GB 单请求被接受而非断连。同时也是解码请求占用两个共享大解码准入槽位的阈值。WebSocket lane 保持固定 100 MiB 表面。" + - key: network.decodedBodyBytes + type: number + default: "10737418240" + env: BILI_DECODED_BODY_BYTES + group: process + desc_en: "Max DECODED bytes after content-encoding expansion (gzip/deflate/br/zstd), checked incrementally during decode (decode-stage 413). Default is 2x the receive budget, mirroring the legacy decode=2x-wire relationship; guards against decompression bombs." + desc_zh: "content-encoding 解压(gzip/deflate/br/zstd)后的解码字节上限,解码过程中增量检查(decode 阶段 413)。默认为接收预算的 2 倍,沿用旧的 decode=2×wire 关系;用于防解压炸弹。" + - key: network.outboundBodyBytes + type: number + default: "5368709120" + env: BILI_OUTBOUND_BODY_BYTES + group: process + desc_en: "Max rebuilt request bytes forwarded upstream (forward-stage 413), counted as UTF-8 bytes of the final body string. Checked before the fetch so an over-budget rebuild never reaches the upstream watchdog allocation." + desc_zh: "转发到上游的重建请求字节上限(forward 阶段 413),按最终 body 字符串的 UTF-8 字节计。在 fetch 之前检查,超预算的重建不会触发上游看门狗分配。" - key: persist type: object default: "{} (all defaults)"