diff --git a/CHANGELOG.md b/CHANGELOG.md index 762936e7dc..65486fbe6e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,6 +10,7 @@ Versions follow the merge of a `*_release-v*` branch; CI publishes to npm on tag - **系统架构文档体系落地(owner 指令 2026-11-08):`docs/architecture/` 成为实时维护的架构枢纽**:新增 `docs/architecture.md` / `docs/architecture.zh-CN.md` 总枢纽(整体架构图中英双版转正 + 8 模块总表),首个样板模块页 `docs/architecture/storage.{md,zh-CN.md}`(统一存储:职责/原理/工作机制/硬不变量,配模块图中英双版)。文件级地图 `reference/architecture.md` 补齐 125 个缺失条目(新存储层 / marker / fastpath / lanes 等全部新模块)并升级为 CI 强制全量覆盖。新增 CI 车道 `architecture-docs`(`tools/check-architecture-docs.mjs`):地图覆盖全 `src/` 文件、总表↔页面成对、图 svg 优先 + png 备选齐备、SVG 基础良构(未转义 `&` 曾让英文图线上破图)。AGENTS.md「Documentation Layout」新增实时维护规则:改结构 ⇒ 同 PR 更新页 + 图 + 总表。其余 7 个模块页(pipeline / lanes / kernel / identity / fastpaths / webui / update)同一 PR 全部补齐——总表 8 行全 ✅(每页 = 中英双版文档 + 中英双版图)。 ### Fixed +- **会话历史的「条数」维度与超大未压缩请求:按真实维度拦下,判死信息点名超重字段(#5000 #5794 #5885)**:#5000 的 26,903 条消息会话(55.2MiB)穿过了**所有**字节/token 闸门、在上游死于消息条数上限——因为条数这个维度从来没有任何闸(它无法被任何字节/token 估算表达:每条消息还各带 ~69B 的 m-ref 标记,26,903 条 ≈ 1.8MB 纯标记)。现在新增按上游配置的**条数闸**(`network.maxMessages` / `BILI_MAX_MESSAGES`,默认 0=关闭——不存在全局「通用上限」,条数上限因上游而异):超限直接 413,信息点名 COUNT 维度并给出可执行指引(重启会话或回退到更早的时点;折叠在一轮 preflight 内关不掉 ~20K 条的缺口,所以是纯判死、不做条数折叠),且在 token 触发器**之前**运行——token 达标但条数超标的体(正是 #5000 的形状)也只有这道闸看得见。同时所有超窗判死信息(#5932 的三道计量闸)现在**点名超重的是哪个字段**——按 #554 计量器同源分桶(messages / tools[] / system / other),不再只给一个总数:#5000 的形状会显示 `tools[] ~X`,巨大粘贴/工具结果会显示 `messages ~X`,运维一眼知道该删什么。另新增 **32MiB 解码摄入成本带**(`network.ingestBodyBytes` / `BILI_INGEST_BODY_BYTES`):#5794 的 51.2MiB/29,619 条请求每轮独占事件循环 5.5–7.2s(本机 CPU 54.6% → 747s FATAL 自重启、全员缓存冷),成本带在 JSON 解析扫掠**之前**按解码后字节 413(stage=ingest)——任何可解析的**文本为主**的 32MiB+ 体都已是 ≥~8M token,超出所有真实模型窗口,放行必然烧钱再死。带图片的体在所有车道豁免(base64 像素按设计不进 token 计量——#496/#1800/#857「图片单独计费、绝不在原始字节上判死」;#3909 的 ~48MiB 图片密集 chat 会话与 #3701 的 ~38.5MiB WS 截图体都是受支持工作负载,由槽位预算/收件上限/车道自身的预发送字节预算+丢弃逻辑治理;Responses 车道无论如何整体豁免);并发槽位预算与 5GiB 收件上限继续独立生效。回归:tests/issue5000-count-gate(3 e2e:token 达标+条数超标→413 零上游 / 无 model 路径 / 条数达标照常转发)、tests/issue5794-ingest-cap(3 e2e:超带文本→413 stage ingest / 带内照常 / 超带带图不拦)、tests/side-request-isolation 分桶计量单测;既有 332 用例(含 #3701/#857/#496/#4079/#4185/#2490 全部回归面)全绿。 - **The log/publish secret scrubber now catches JSON-quoted credentials and Groq/Hugging Face keys.** `redactSecretsInText` (every log line, and via `redactForPublish` the diagnostics blob that lands in public GitHub issues) required `:`/`=` right after the key name, so the JSON/dict shape upstream error bodies and config dumps use — `"api_key": "…"`, `"password": "…"`, `'access_token': '…'` — slipped through unless the value happened to start with a known prefix such as `sk-`. An optional quote after the key name now matches it; `gsk_…` (Groq) and `hf_…` (Hugging Face) keys, both on the public-host list, are masked like `sk-…`. Usage-count keys (`"prompt_tokens"`, `"tokens"`, `"max_tokens"`) stay verbatim. Regression: `tests/log-mask.test.ts`. - **超窗请求不再裸发上游:三道转发闸按「真实出站体」计量(#5000 #4543 #5885),未压缩大请求纳入 32MiB 成本带准入(#5794)**:旧判据只量「消息核心文本的字节」,看不见 system/tools/tool_calls 参数——#5000 的 55.2MiB 体(84 个工具)与 #4543 的 4.1MiB CJK 体(1,121,674 tok vs 1M 窗口)都能让全部闸门沉默,然后「转发一次取证」把必然 400 的体真发出去烧钱。现在三个「转发仲裁点」(压缩后再检、#1839 仲裁臂、出站熔断)改按出站体的**逐字符 token 计量**(system+tools+arguments+messages,CJK 感知,#554 侧车道同源尺子)对照**真实模型窗口**本地 413,不再裸发;取证探针闸保持 #2490 字节契约不动(那里的一次 400 是设计内的证据回路,见 #2313/#1001);图片体不进判死——计量器补排 responses 车道的 `image_url` 键(此前把截图 base64 当文本量,40MB 截图体被量成 ~10M token,误杀 #3701 的 WS shed 重发与 #857 的图片计费仲裁转发),判死只比文本量,图片单独超窗仍走 #496/#1800 上游计费仲裁。另:成本准入旧阈值 = 5GiB 收件上限,2 槽位预算形同虚设,#5794 的 51.2MiB/轮未压缩请求(每轮独占事件循环 5.5–7.2s、占本机 CPU 54.6% → 747s FATAL 自重启)从此被拦;未压缩体现在按 32MiB 成本带准入(运维显式调低收件上限仍逐字节生效)。tests:issue4079/issue4185/request-body-budget 新增实测档 e2e 与 32MiB 顺序用例;既有 89+67 用例全绿,#857/#3701/#496/图片与计量相关 173 用例回归全绿。#5885 的 modelContextLimit 被忽略属独立缺陷,另行跟进。 - **dsh 闸门:必然被上游拒绝的超窗请求不再裸发直连(#5613 安全网 ①②③)**:`none`/`threw` 归属车道(如经网关的 ds-harness-remote)上的超窗历史请求,旧行为二选一——要么按乐观的 bytes/4 估算判「不超窗」而直接裸发上游(实测计费偏差≈×1.14,估算系统性偏低;真实现场把 max_tokens 挤到 1、乃至上游 400 后整会话报废),要么即便超窗也没有任何持续留痕(只记第一行之后全静默)。现在双档计价:原有 bytes/4 下限继续标记「必然超窗」,另加 ×1.14 校准估算档(会话 ledger 实测偏差,常数 `CALIBRATED_TEXT_TOKEN_RATIO`,src/util.ts),过线即本地 422 拒绝而不是骑 #1117 去赌上游——小体量的合法直通语义(sidecar 短调用等)逐字节保留,未知尺寸流式体保持 fail-open。每次裸发 DIRECT 现在都带体积(`this body N B`)或「尺寸未知、超窗熔断未跑」明示,按 首次/状态变化/每第10次 落 durable 日志(bili.log,[dsh-client] 通道)——连续泄漏环路从此必有痕。tests/dsh-native.test.ts:#2924/#4068 双臂升级为双档断言+校准边界钉(3,508,768 B 直通 / 3,508,772 B 拒)、新增留痕节拍回归、#4111/#4912 谓词改名对齐。 diff --git a/docs/configuration.md b/docs/configuration.md index fad61fe8fc..2bee3edc0f 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -182,6 +182,8 @@ This index is generated from `website/config-reference/*.yaml` — edit the seed | `network.outboundBodyBytes` | number | 5368709120 | BILI_OUTBOUND_BODY_BYTES | Max rebuilt request bytes forwarded upstream (forward-stage 413), counted as UTF-8 bytes of the final body string. Checked before the fetch so an over-budget rebuild never reaches the upstream watchdog allocation. | | `network.wsOutboundBytes` | number | 10485760 | BILI_WS_OUTBOUND_BYTES | Soft byte budget for the WebSocket lane only (#3701): when the final serialized Responses WS frame exceeds it, older images are shed from the OUTBOUND COPY first (archived as placeholder notes — originals stay in session storage, recoverable via read_original_image / decompress on tool-armed lanes — codex-style lanes that reject injected tools keep the size-only annotation), then whatever remains is boundedly downscaled through the jimp maxDim ladder. If the frame still does not fit and the upstream closes with 1009 before any event, one strictly smaller resend is attempted within the same request. Set 0 to disable shedding (previous send-as-is behavior). HTTP lanes are unaffected — the hard refusal cap stays network.outboundBodyBytes. | | `network.httpOutboundBytes` | number | 10485760 | BILI_HTTP_OUTBOUND_BYTES | Soft byte budget for the plain/proxy/pi-native HTTP relay lane (#3909): when the final serialized request exceeds it, older images are shed from the OUTBOUND COPY first (archived as placeholder notes — originals stay in session storage, recoverable via read_original_image / decompress on tool-armed lanes — codex-style lanes that reject injected tools keep the size-only annotation), then whatever remains is boundedly downscaled through the jimp maxDim ladder. Closes the gap image-dense sessions leave open: token-space folding decides in tokens while the upstream rejects in bytes / pixel-decode CPU (Cloudflare Worker 503), so a ~48 MiB request 503s forever with every retry resending the same pixels. Applies only when no managed transport is installed (the spawned relay); the WS lane uses network.wsOutboundBytes and managed HTTP transports (inline-engine / dsh-inline) are never shed here (#3701 Finding 1). Set 0 to disable (previous send-as-is); the hard refusal cap stays network.outboundBodyBytes. | +| `network.ingestBodyBytes` | number | 33554432 | BILI_INGEST_BODY_BYTES | Cost band (decoded bytes) for the JSON.parse + normalization sweep AFTER decode (#5794 item 2). A TEXT-DOMINATED body above it is refused with an actionable 413 (stage ingest) before the sweep, because any parseable text body over 32 MiB is already ~8M tokens — beyond every real model window — and burning seconds of event-loop time to parse it (the 51.2 MiB / 29,619-message incident: 5.5–7.2 s per round, 54.6% of local CPU, 747 s FATAL) cannot end in success. Image-bearing bodies are exempt on every lane: base64 pixels are excluded from the token meter by design (images billed separately, never doomed on raw bytes — #496/#1800/#857), so over-band image mass stays governed by the admission slots, the receive cap and each lane's own shed / pre-send byte budget (#3909 ~48 MiB image-dense sessions, #3701 WS screenshots; the Responses lane is exempt outright regardless). Distinct from the admission SLOTS (network.receiveBodyBytes threshold): the slots cap CONCURRENCY of large decodes, this cap bounds a single body. Set 0 to disable (legacy parse-everything). | +| `network.maxMessages` | number | 0 | BILI_MAX_MESSAGES | Per-upstream hard cap on the messages[] array length (#5000 item 2) — the COUNT dimension: a wire constraint no byte/token estimate expresses (the 26,903-message / 55.2 MiB #5000 session cleared every size gate and died at the upstream with the count error). Per-UPSTREAM on purpose: there is no universal value, so the default is 0 = disabled; set it to your upstream's known limit and over-count bodies get an actionable 413 (restart the session or rewind) instead of a guaranteed upstream 400. Count only — it never folds. Enforced in preflight (before the token trigger) and at the egress fuse for the no-model lane. | | `network.caFile` | string | unset | BILI_CA_FILE | Path to a PEM file with extra root CAs trusted by bili's OWN upstream fetches (#3776). Merged with Node's built-in roots and any NODE_EXTRA_CA_CERTS entries — never replaces them. For security software doing HTTPS/encrypted-connection scanning (Kaspersky, enterprise DLP) or a corporate MITM proxy that re-signs the upstream: export that root CA to PEM and point this at it. Unset = default trust, byte-identical behavior. | | `guardrails` | object | {} (all defaults) | — | Guardrails: tool-output guardrails for the pi/omp plugin bundles (#3066) plus the proxy-side text-run stream guard (#3406). | | `guardrails.bashTimeout` | number | 60 | BILI_GUARDRAIL_BASH_TIMEOUT | Seconds injected as the bash tool's timeout when the model omits it (#3066 owner intent, re-affirmed #3366: an unsupervised bash call could hang a pi session for a whole day — the host schema provides no default). Set N to tune, 0 to disable injection (pi-native no-default). | @@ -1916,9 +1918,11 @@ File keys resolve only when the matching env var is unset. Defaults in parenthes | `BILI_IMAGE_BILLING` | `imageBilling` | auto (resolves to pixels) | | `BILI_IMAGE_RESCALE` | `imageRescale` | true | | `BILI_IMAGE_TOKEN_CAP` | `imageTokenCap` | unset (uncapped) | +| `BILI_INGEST_BODY_BYTES` | `network.ingestBodyBytes` | 33554432 | | `BILI_INVARIANT_JUDGE` | `diagnostics.invariantJudge` | true | | `BILI_KEEP_ALIVE_TIMEOUT_MS` | `network.keepAliveTimeoutMs` | 5000 | | `BILI_LOG_MASK_HOSTS` | `maskHosts` | true | +| `BILI_MAX_MESSAGES` | `network.maxMessages` | 0 | | `BILI_MAX_SESSIONS` | `sessions.max` | 256 | | `BILI_MAX_SHRINK_PER_COMPRESS` | `network.maxShrinkPerCompress` | unset | | `BILI_MCP_PROXY` | `native.attachUrl` | unset (discover per lane) | diff --git a/docs/configuration.zh-CN.md b/docs/configuration.zh-CN.md index 7461448878..04e49b2e55 100644 --- a/docs/configuration.zh-CN.md +++ b/docs/configuration.zh-CN.md @@ -182,6 +182,8 @@ | `network.outboundBodyBytes` | number | 5368709120 | BILI_OUTBOUND_BODY_BYTES | 转发到上游的重建请求字节上限(forward 阶段 413),按最终 body 字符串的 UTF-8 字节计。在 fetch 之前检查,超预算的重建不会触发上游看门狗分配。 | | `network.wsOutboundBytes` | number | 10485760 | BILI_WS_OUTBOUND_BYTES | 仅针对 WebSocket 通道的软字节预算(#3701):当最终序列化的 Responses WS 帧超出该预算时,优先从出站副本中裁减较早的图片(归档为占位符备注——原始文件仍保留在会话存储中,可通过 read_original_image / decompress 恢复——仅限注册了 bili 工具的通道;拒绝注入工具的 codex 类通道只有尺寸备注),随后对剩余内容通过 jimp maxDim 阶梯进行有界降采样。若帧仍超出限制且上游在任何事件发出前即以 1009 关闭连接,则在同一请求内尝试一次严格更小的重发。设为 0 可禁用裁减(即恢复此前的原样发送行为)。HTTP 通道不受影响——硬性拒绝上限仍为 network.outboundBodyBytes。 | | `network.httpOutboundBytes` | number | 10485760 | BILI_HTTP_OUTBOUND_BYTES | 面向普通/代理/pi 原生 HTTP 中继通道的软字节预算(#3909):当最终序列化的请求超出该预算时,优先从出站副本中裁减较早的图片(归档为占位符备注——原始文件仍保留在会话存储中,可通过 read_original_image / decompress 恢复——仅限注册了 bili 工具的通道;拒绝注入工具的 codex 类通道只有尺寸备注),随后对剩余内容通过 jimp maxDim 阶梯进行有界降采样。它补上图片密集会话留下的缺口:token 空间折叠按 token 决策,而上游按字节/像素解码 CPU 拒收(Cloudflare Worker 503),于是 ~48 MiB 的请求会永远 503、每次重试都重发同样的像素。仅在未安装受管传输(即 spawn 出来的中继)时生效;WebSocket 通道用 network.wsOutboundBytes,受管 HTTP 传输(inline-engine / dsh-inline)在此从不裁减(#3701 Finding 1)。设为 0 可禁用(恢复此前的原样发送);硬性拒绝上限仍为 network.outboundBodyBytes。 | +| `network.ingestBodyBytes` | number | 33554432 | BILI_INGEST_BODY_BYTES | 解码之后进入 JSON.parse+归一化清扫的成本带(解码后字节,#5794 第 2 项)。超限的**文本为主**的体在清扫之前以可操作的 413(stage ingest)拒收:任何可解析的 32 MiB+ 文本体都 ≥ ~8M token、超过一切真实模型窗口,为它烧掉数秒事件循环时间没有意义(51.2 MiB / 29,619 条消息现场:每轮 5.5–7.2s、占本机 CPU 54.6%、747s FATAL)。带图片的体在所有车道豁免:base64 像素按设计不进 token 计量(图片单独计费、绝不在原始字节上判死——#496/#1800/#857),超带的图片体量仍由准入槽位、收件上限与各车道自身的裁减/预发送字节预算治理(#3909 ~48 MiB 图片密集会话、#3701 WS 截图;Responses 车道无论如何整体豁免)。与大解码准入槽位(network.receiveBodyBytes 阈值)不同:槽位限的是大解码的并发数,本上限限的是单个体的成本。设为 0 可禁用(恢复此前的全解析)。 | +| `network.maxMessages` | number | 0 | BILI_MAX_MESSAGES | 单上游 messages[] 数组长度硬限(#5000 第 2 项)——COUNT 维度:任何字节/token 估算都表达不了的线路约束(#5000 的 26,903 条消息 / 55.2 MiB 会话清掉所有尺寸闸后仍在上游以条数错误死掉)。刻意按上游设置:没有通用值,默认 0 = 关闭;把它设为你所连上游的已知上限后,超条数的体会得到可操作的 413(重启会话或 rewind)而不是一次必然的上游 400。只判条数、从不折叠。在 preflight(token 触发器之前)与无模型车道的出站熔断处强制执行。 | | `network.caFile` | string | unset | BILI_CA_FILE | PEM 文件路径,包含要额外信任的根 CA(#3776),仅作用于 bili 自己的上游 fetch。与 Node 内置根证书及 NODE_EXTRA_CA_CERTS 中的条目合并——从不替换。安全软件做 HTTPS/加密连接扫描(卡巴斯基、企业 DLP)或企业 MITM 代理重签上游时使用:把该厂商根证书导出为 PEM 并指向此键。未设置 = 默认信任,行为逐字节不变。 | | `guardrails` | object | {} (all defaults) | — | 守卫集合:pi/omp 插件的工具输出守卫(#3066)及代理侧流式 text-run 守卫(#3406)。 | | `guardrails.bashTimeout` | number | 60 | BILI_GUARDRAIL_BASH_TIMEOUT | 模型未写 timeout 时注入到 bash 工具的秒数(#3066 owner 有意为之,#3366 重申:无人看管的 bash 会把 pi 会话挂死一整天——宿主 schema 本身无默认值)。设 N 调整秒数,设 0 关闭注入(恢复 pi 原生行为)。 | @@ -1893,9 +1895,11 @@ ACP 原生 agent(当前为 `pi` 扩展)会在每个进程内向代理上报 | `BILI_IMAGE_BILLING` | `imageBilling` | auto (resolves to pixels) | | `BILI_IMAGE_RESCALE` | `imageRescale` | true | | `BILI_IMAGE_TOKEN_CAP` | `imageTokenCap` | unset (uncapped) | +| `BILI_INGEST_BODY_BYTES` | `network.ingestBodyBytes` | 33554432 | | `BILI_INVARIANT_JUDGE` | `diagnostics.invariantJudge` | true | | `BILI_KEEP_ALIVE_TIMEOUT_MS` | `network.keepAliveTimeoutMs` | 5000 | | `BILI_LOG_MASK_HOSTS` | `maskHosts` | true | +| `BILI_MAX_MESSAGES` | `network.maxMessages` | 0 | | `BILI_MAX_SESSIONS` | `sessions.max` | 256 | | `BILI_MAX_SHRINK_PER_COMPRESS` | `network.maxShrinkPerCompress` | unset | | `BILI_MCP_PROXY` | `native.attachUrl` | unset (discover per lane) | diff --git a/docs/lifecycle.md b/docs/lifecycle.md index 35bb813ec8..45e75b30a2 100644 --- a/docs/lifecycle.md +++ b/docs/lifecycle.md @@ -113,6 +113,8 @@ | `BILI_OUTBOUND_BODY_BYTES` | network.outboundBodyBytes,5 GiB——重建后转发上游的硬上限(#3128) | | `BILI_WS_OUTBOUND_BYTES` | network.wsOutboundBytes,10 MiB——WS 车道出站软预算:超限先把旧的非保护图归档/降采样再发送(#3701:网关对大帧 close 1009;0 关闭预发送削减;HTTP 车道不受影响) | | `BILI_HTTP_OUTBOUND_BYTES` | network.httpOutboundBytes,10 MiB——普通/代理/pi 原生 HTTP 中继车道出站软预算:超限先把旧的非保护图归档/降采样再发送(#3909:图片密集会话按字节/像素解码 CPU 被上游拒收、token 折叠不触发;仅在无受管传输时生效,受管 HTTP/dsh-inline 从不裁减 #3701 Finding 1;0 关闭) | +| `BILI_INGEST_BODY_BYTES` | network.ingestBodyBytes,32 MiB——解码后进入 JSON.parse+归一化清扫的成本带(#5794:51.2 MiB 体每轮吞 5.5–7.2s 事件循环 → 747s FATAL;超带的**文本为主**体即 413,因为任何可解析的 32 MiB+ 文本体 ≥ ~8M token、超过一切真实窗口;带图片的体豁免——像素按设计不进 token 计量,由槽位预算/收件上限/车道 shed 治理(#496/#3909/#3701);0 关闭) | +| `BILI_MAX_MESSAGES` | network.maxMessages,默认 0(关闭)——单上游 messages[] 数组长度硬限(COUNT 维度,任何字节/token 估算都表达不了;#5000:26,903 条消息体清掉所有尺寸闸仍在上游死掉;超限 413 提示 restart/rewind,不做折叠;按上游各自设置) | | `BILI_CACHE_KEEPALIVE`(#3376) | unset(false;同 `cacheKeepalive.enabled` 层级 env 覆盖) | | `BILI_CACHE_KEEPALIVE_ASSUME_TTL` | unset(从 body cache_control 断点探测) | | `BILI_CACHE_KEEPALIVE_TTL_FACTOR` | 0.9 | diff --git a/src/config.ts b/src/config.ts index 82424f4587..e84e88d686 100644 --- a/src/config.ts +++ b/src/config.ts @@ -2321,6 +2321,17 @@ type FileConfig = { * SELF_SIGNED_CERT_IN_CHAIN. MERGED with Node's built-in roots, never replacing them. * Env BILI_CA_FILE wins. */ caFile?: string; + /** #5000 item 2: per-upstream hard cap on the messages[] array length — a wire constraint no + * byte/token estimate expresses (the 26,903-message #5000 session cleared every size gate and + * died at the upstream). Per-UPSTREAM on purpose: default 0 = disabled; set it to your upstream's + * known limit and over-count bodies get an actionable 413 (restart / rewind) instead of a + * guaranteed upstream 400. Count only — it never folds. Env BILI_MAX_MESSAGES wins. */ + maxMessages?: number; + /** #5794 item 2: cost band (decoded bytes) for the JSON.parse + normalization sweep AFTER + * decode; a body above it is refused with an actionable 413 before the sweep, because any + * parseable body over 32 MiB is ≥ ~8M tokens — beyond every real model window. 0 disables + * the ingest cap (legacy parse-everything). Env BILI_INGEST_BODY_BYTES wins. */ + ingestBodyBytes?: number; }; /** Session persistence knobs (#2030) — was BILI_PERSIST_* env-only. */ persist?: { diff --git a/src/content-encoding.ts b/src/content-encoding.ts index f1a39eab42..848856e47a 100644 --- a/src/content-encoding.ts +++ b/src/content-encoding.ts @@ -14,6 +14,21 @@ export class DecompressedTooLargeError extends Error { } } +/** #5794 item 2: raised for a decoded body above the INGEST cost band + * (network.ingestBodyBytes / BILI_INGEST_BODY_BYTES, default 32 MiB). Distinct + * from DecompressedTooLargeError (a decode-cap bomb guard): this one bounds the + * JSON.parse + normalization sweep itself — a 51.2 MiB / 29,619-message body + * spent 5.5–7.2 s per round there (54.6% of local CPU → 747 s event-loop + * FATAL → every session's cache went cold). Any parseable body above the band + * is already ≥ ~8M tokens, beyond every real model window, so refusing it is + * free. The message is the operator-facing actionable guidance. */ +export class IngestBodyTooLargeError extends Error { + constructor(public readonly limit: number, public readonly bytes: number) { + super(`request body ${bytes.toLocaleString("en-US")} B (after decode) exceeds the ingestion cost band ${limit.toLocaleString("en-US")} B (#5794: bodies this large are ≥ ~8M tokens — beyond every model window — and burn the event loop for seconds in the parse sweep). Reduce the session history — restart the session or rewind to an earlier point — and retry.`); + this.name = "IngestBodyTooLargeError"; + } +} + /** Stream-decompress under a hard byte cap: rejects with DecompressedTooLargeError * the instant output exceeds `max`, so a bomb never inflates in memory and the * cap is enforced deterministically per codec. node's one-shot `maxOutputLength` diff --git a/src/knobs.ts b/src/knobs.ts index e7054439ef..2586bf3974 100644 --- a/src/knobs.ts +++ b/src/knobs.ts @@ -283,6 +283,12 @@ const DEFAULT_LIMITS_DECODED_BYTES = 10 * 1024 * 1024 * 1024; const DEFAULT_LIMITS_OUTBOUND_BYTES = 5 * 1024 * 1024 * 1024; const DEFAULT_WS_OUTBOUND_BYTES = 10 * 1024 * 1024; const DEFAULT_HTTP_OUTBOUND_BYTES = 10 * 1024 * 1024; +// #5794: the ingest cost band (decoded bytes) that may run the JSON.parse / +// normalization sweep. 32 MiB ≈ the observed 51.2 MiB/29,619-msg incident +// (5.5–7.2 s per round, 54.6% of local CPU, 747 s event-loop FATAL); any +// parseable body ABOVE it is already ≥ ~8M tokens = 5–10× the largest model +// window, so it can never succeed. 0 = no ingest cap (legacy). +const DEFAULT_INGEST_BODY_BYTES = 32 * 1024 * 1024; /** BILI_RECEIVE_BODY_BYTES > network.receiveBodyBytes > 5 GiB. Max inbound wire bytes * read into memory per request. #3128: these were hardcoded 100 MiB constants; the @@ -336,6 +342,29 @@ export function limitsHttpOutboundBytes(): number { return Number.isFinite(v) && v >= 0 ? Math.floor(v) : DEFAULT_HTTP_OUTBOUND_BYTES; } +/** BILI_INGEST_BODY_BYTES > network.ingestBodyBytes > 32 MiB. Cost band for the + * JSON.parse + normalization sweep AFTER decode (the 51.2 MiB #5794 body spent + * seconds there per round). Bodies above it are refused before the sweep with + * an actionable 413; any parseable body over 32 MiB is ≥ ~8M tokens, i.e. + * beyond every real model window, so nothing valid is blocked. 0 disables the + * ingest cap (legacy parse-everything). */ +export function limitsIngestBytes(): number { + const v = tInt("BILI_INGEST_BODY_BYTES", loadConfigFile().network?.ingestBodyBytes); + return Number.isFinite(v) && v >= 0 ? Math.floor(v) : DEFAULT_INGEST_BODY_BYTES; +} + +// #5000 item 2: per-upstream hard cap on the messages[] array length — a wire +// constraint no byte/token estimate expresses (the 26,903-message #5000 session +// cleared every size gate and died at the upstream). Per-UPSTREAM on purpose: +// there is no universal value, so the default is 0 = disabled; operators set +// BILI_MAX_MESSAGES / network.maxMessages to their upstream's known limit and +// over-limit bodies get an actionable 413 (restart / rewind) instead of a +// guaranteed upstream 400. Count only — it never folds. +export function limitsMaxMessages(): number { + const v = tInt("BILI_MAX_MESSAGES", loadConfigFile().network?.maxMessages); + return Number.isFinite(v) && v > 0 ? Math.floor(v) : 0; +} + // --- session persistence (was BILI_PERSIST_* env-only) --- function filePersist() { diff --git a/src/preflight.ts b/src/preflight.ts index 05d38180bf..6c6603638c 100644 --- a/src/preflight.ts +++ b/src/preflight.ts @@ -469,6 +469,53 @@ export function estimateRawBodyTokens(parsed: unknown): number { return tokens; } +/** #5000 item 3: which DIMENSION carries the outbound mass. The doom messages + * must say whether it is messages[] (a giant paste / tool result), tools[] + * (84 schemas in #5000), the system prompt, or structural other — "trim the + * oversized content" is actively misleading for a 55.2 MiB body whose + * messages[] is only 1.44 MiB. Same walk + same exclusions as + * estimateRawBodyTokens, split on the TOP-LEVEL field, so `total` always + * equals estimateRawBodyTokens(parsed) exactly. */ +export interface RawBodyBreakdown { + total: number; + messages: number; + tools: number; + system: number; + other: number; +} + +export function estimateRawBodyBreakdown(parsed: unknown): RawBodyBreakdown { + const buckets: [number, number, number, number] = [0, 0, 0, 0]; // messages, tools, system, other + const walk = (value: unknown, key: string | undefined, bucket: number): void => { + if (typeof value === "string") { + if (!key || !NON_TEXT_BODY_KEYS.has(key)) buckets[bucket] += defaultCountTokens(value); + return; + } + if (Array.isArray(value)) { + for (const item of value) walk(item, key, bucket); + return; + } + if (value && typeof value === "object") { + for (const [k, v] of Object.entries(value as Record)) walk(v, k, bucket); + } + }; + if (parsed && typeof parsed === "object") { + for (const [k, v] of Object.entries(parsed as Record)) { + const bucket = k === "messages" ? 0 : k === "tools" ? 1 : k === "system" ? 2 : 3; + walk(v, k, bucket); + } + } else { + walk(parsed, undefined, 3); + } + return { + total: buckets[0] + buckets[1] + buckets[2] + buckets[3], + messages: buckets[0], + tools: buckets[1], + system: buckets[2], + other: buckets[3], + }; +} + // #553: upper-bound variant of estimateCoreMessages — every character counts // as one token. A BPE token covers >=1 char (Latin/code) and CJK is already // ~1 token/char, so this never undershoots the real count, unlike diff --git a/src/server.ts b/src/server.ts index 8229f2cb0a..0646cf5f0c 100644 --- a/src/server.ts +++ b/src/server.ts @@ -84,7 +84,7 @@ import { auditStartupSummaryCredentials } from "./external-summary-credentials.j import { applyRanges } from "./stream.js"; import { attachSubagentSessions } from "./subagent-sessions.js"; import { buildSessionCacheReport, handleAcpCache, learnedImageReserve, noteClientAbort, readKeySwitchStats, readModelSwitchStats, settleUsageReport } from "./cache-ledger.js"; -import { extractBillingAttributionBlock, extractSummaryFromSse, preflightCompress, estimateCoreMessages, estimateCoreMessagesUpper, estimateCoreMessagesUpperBytes, describeCoreTextDuplication, estimateRawBodyTokens, type PreflightResult } from "./preflight.js"; +import { extractBillingAttributionBlock, extractSummaryFromSse, preflightCompress, estimateCoreMessages, estimateCoreMessagesUpper, estimateCoreMessagesUpperBytes, describeCoreTextDuplication, estimateRawBodyTokens, estimateRawBodyBreakdown, type PreflightResult, type RawBodyBreakdown } from "./preflight.js"; import { buildDecisionPrompt, buildDirectiveText, consumeFallback, DEFAULT_DECIDE_MAX_TOKENS, DECIDE_TIMEOUT_MS, extractDecisionText, ladderMode, parseDecision, recordDecision, resolveDecisionRange, type DecideConfig, type DecisionOutcome } from "./nudge-decide.js"; import { gcConfigFromEnv, gcSessionFiles } from "./session-gc.js"; import { countImagesInParsedBody, countImagesInRawBody, imageTokensInRawBody, imageTokensInParsedBody, resolveImageBilling, upstreamHost, type ResolvedImageBilling } from "./image-tokens.js"; @@ -140,6 +140,8 @@ import { keepAliveTimeoutMs as knobKeepAliveTimeoutMs, keepResponseId as knobKeepResponseId, limitsReceiveBytes as knobLimitsReceiveBytes, + limitsMaxMessages, + limitsIngestBytes as knobLimitsIngestBytes, noCompressPrompt as knobNoCompressPrompt, noInjectTool as knobNoInjectTool, postResponseLingerMs as knobPostResponseLingerMs, @@ -3746,7 +3748,20 @@ export function buildWindowShrinkNote(args: { // tokens are charged separately by the caller (they sit in binary-carrying // keys the walk excludes by design). Unparseable bodies return undefined — // callers keep the byte-floor fallback for them. -export function measuredOutboundTokens(prepared: { body: string | Buffer; parsedBody?: unknown }): number | undefined { +interface MeasuredOutbound { + tokens: number; + breakdown: RawBodyBreakdown; +} + +// #5000 item 3: dimension clause for the measured-doom messages — which part +// of the body carries the mass, so "trim the oversized content" tells the +// operator WHERE (a 55.2 MiB #5000 body with 1.44 MiB of messages[] means the +// mass is tools[], not a giant paste). +export function measuredBreakdownClause(b: RawBodyBreakdown): string { + return `messages ~${b.messages.toLocaleString("en-US")} · tools[] ~${b.tools.toLocaleString("en-US")} · system ~${b.system.toLocaleString("en-US")} · other ~${b.other.toLocaleString("en-US")}`; +} + +export function measuredOutbound(prepared: { body: string | Buffer; parsedBody?: unknown }): MeasuredOutbound | undefined { let parsed: unknown = prepared.parsedBody; if (parsed === undefined) { const raw = typeof prepared.body === "string" ? prepared.body : prepared.body.toString("utf8"); @@ -3756,7 +3771,8 @@ export function measuredOutboundTokens(prepared: { body: string | Buffer; parsed return undefined; } } - return estimateRawBodyTokens(parsed); + const breakdown = estimateRawBodyBreakdown(parsed); + return { tokens: breakdown.total, breakdown }; } export async function preflightCompressIfNeeded( @@ -3960,6 +3976,30 @@ export async function preflightCompressIfNeeded( const billed = session.stats.lastInputTokens > 0 && session.stats.lastInputTokensSource === "usage" ? session.stats.lastInputTokens : "none"; log("debug", `[${session.id}] [usage-vs-est] upstream-billed=${billed} est=${Math.round(textEstimate)} k̂=${kFactor !== undefined ? kFactor.toFixed(2) : "n/a"} route=${currentOrigin === undefined ? "?" : maskUrlForLog(currentOrigin)} model=${model ?? "?"} baseline-grade=${session.stats.lastInputTokensSource ?? "none"} (#2283)`); } + // #5000 item 2: the COUNT dimension — a hard upstream wire constraint on + // the messages[] array length that no byte/token estimate can express. + // #5000's 55.2 MiB / 26,903-message session cleared EVERY size gate (core + // text 1.44 MiB, 2 MiB absolute floor, egress byte floor) and died at the + // upstream with the count error. The limit is per-upstream: operators opt + // in with BILI_MAX_MESSAGES (or network.maxMessages); 0 = disabled (no + // global default — a wrong value must not false-block lanes whose + // upstream has no count limit). Doom-only: folding cannot close a + // 21,903-message gap inside one preflight round budget (the planner is + // token-benefit-driven, #3444), and the actionable remedy is client-side + // (restart / rewind). Runs BEFORE the trigger guard: a token-fit + // 6,000-message body never triggers preflight yet still 400s upstream — + // that IS the #5000 case. + const maxMessages = limitsMaxMessages(); + if (maxMessages > 0) { + const msgCount = prepared.processedMessages.length > 0 ? prepared.processedMessages.length : prepared.originalMessages.length; + if (msgCount > maxMessages) { + const gap = msgCount - maxMessages; + const countMessage = `message count ${msgCount.toLocaleString("en-US")} exceeds the upstream wire limit ${maxMessages.toLocaleString("en-US")} (COUNT dimension — the messages[] array length is a hard upstream constraint that no byte/token estimate expresses, #5000: a 26,903-message session cleared every size gate and died at the upstream). Folding cannot close ~${gap.toLocaleString("en-US")} messages inside one preflight; reduce the session history — restart the session or rewind to an earlier point — and retry.`; + log("error", `[${session.id}] preflight fail-fast 413 (retryable=false, count gate): ${countMessage}`); + noteLocalRefusal(session, currentOrigin, model, limit); + return { failFast: true, status: 413, message: countMessage, retryable: false, respond: !res.writableEnded }; + } + } if (limit <= 0 || !model || !triggerFires) return prepared; // #3846: an earlier local 413 for THIS origin+model at an equal or wider // window is independent evidence the payload cannot fit here — the @@ -4394,9 +4434,11 @@ export async function preflightCompressIfNeeded( // fallbacks stay binding for those bodies. let outboundBudgetTokens: number | undefined; let outboundRawTokens: number | undefined; + let outboundBreakdown: RawBodyBreakdown | undefined; try { const outBody: unknown = JSON.parse(typeof outbound.body === "string" ? outbound.body : outbound.body.toString("utf8")); - outboundRawTokens = estimateRawBodyTokens(outBody); + outboundBreakdown = estimateRawBodyBreakdown(outBody); + outboundRawTokens = outboundBreakdown.total; const outField = outputBudgetField(outBody); const rawBudget = outField !== null ? readOutputBudget(outBody as Record, outField) : undefined; if (typeof rawBudget === "number" && rawBudget > 0) outboundBudgetTokens = rawBudget; @@ -4432,7 +4474,7 @@ export async function preflightCompressIfNeeded( if (outboundRawTokens !== undefined && outboundRawTokens > armMeasuredBar) { const outMsgs = outbound.processedMessages.length > 0 ? outbound.processedMessages : outbound.originalMessages; log("error", `[${session.id}] preflight 413 measured-body profile (post-fold): ${describeCoreTextDuplication(outMsgs)}`); - const measuredDoomMessage = `rebuilt outbound body measures ~${outboundRawTokens.toLocaleString("en-US")} TEXT tokens (system + tools + arguments + messages; CJK-aware count, the #554 guard meter; images billed separately, never doom here — #496/#1800/#857) vs the model window ${armMeasuredBar.toLocaleString("en-US")} (model=${model}); the body ITSELF exceeds the window, so it was NOT forwarded for upstream arbitration. Remove or trim the oversized content (e.g. a giant paste or tool result) and retry.`; + const measuredDoomMessage = `rebuilt outbound body measures ~${outboundRawTokens.toLocaleString("en-US")} TEXT tokens (${outboundBreakdown ? measuredBreakdownClause(outboundBreakdown) : "by field: n/a"}; CJK-aware count, the #554 guard meter; images billed separately, never doom here — #496/#1800/#857) vs the model window ${armMeasuredBar.toLocaleString("en-US")} (model=${model}); the body ITSELF exceeds the window, so it was NOT forwarded for upstream arbitration. Remove or trim the oversized dimension named above (e.g. a giant paste or tool result in messages[], or tools[] schemas) and retry.`; log("error", `[${session.id}] preflight fail-fast 413 (retryable=false): ${measuredDoomMessage}`); noteLocalRefusal(session, currentOrigin, model, limit); return { failFast: true, status: 413, message: measuredDoomMessage, retryable: false, respond: !res.writableEnded }; @@ -4531,7 +4573,7 @@ export async function preflightCompressIfNeeded( const gateMeasuredBar = typeof resolvedNativeWindow === "number" && resolvedNativeWindow > 0 ? resolvedNativeWindow : limit; if (outboundRawTokens !== undefined && outboundRawTokens > gateMeasuredBar) { log("error", `[${session.id}] preflight 413 measured-body profile: ${describeCoreTextDuplication(outMsgs)}`); - const measuredDoomMessage = `outbound body measures ~${outboundRawTokens.toLocaleString("en-US")} TEXT tokens even after preflight compression (system + tools + arguments + messages; CJK-aware count, the #554 guard meter; images billed separately, never doom here — #496/#1800/#857) vs the model window ${gateMeasuredBar.toLocaleString("en-US")} (model=${model}); the body ITSELF exceeds the window, so it was NOT forwarded. Remove or trim the oversized content (e.g. a giant paste or tool result) and retry.`; + const measuredDoomMessage = `outbound body measures ~${outboundRawTokens.toLocaleString("en-US")} TEXT tokens even after preflight compression (${outboundBreakdown ? measuredBreakdownClause(outboundBreakdown) : "by field: n/a"}; CJK-aware count, the #554 guard meter; images billed separately, never doom here — #496/#1800/#857) vs the model window ${gateMeasuredBar.toLocaleString("en-US")} (model=${model}); the body ITSELF exceeds the window, so it was NOT forwarded. Remove or trim the oversized dimension named above (e.g. a giant paste or tool result in messages[], or tools[] schemas) and retry.`; log("error", `[${session.id}] preflight fail-fast 413 (retryable=false): ${measuredDoomMessage}`); noteLocalRefusal(session, currentOrigin, model, limit); return { failFast: true, status: 413, message: measuredDoomMessage, retryable: false, respond: !res.writableEnded }; diff --git a/src/server/handle.ts b/src/server/handle.ts index 50a3eb38d1..a946add906 100644 --- a/src/server/handle.ts +++ b/src/server/handle.ts @@ -26,7 +26,7 @@ import { storeEffectiveSearchPlanAware } from "../decompress-shared.js"; import { externalSummaryEnabled, messageTimestampSections } from "../external-summary-surface.js"; import { estimateRawBodyTokens } from "../preflight.js"; import { gcDumpDirIfConfigured } from "../state-gc.js"; -import { imageTokensInParsedBody } from "../image-tokens.js"; +import { countImagesInRawBody, imageTokensInParsedBody } from "../image-tokens.js"; import { APIG_RESIGN_CREDENTIAL_HEADER, APIG_RESIGN_HEADER, APIG_RESIGN_SCHEME, clearSignedRefusal, decodeApigCredential, inboundSignedScheme, recordSignedRefusal, signedRefusal } from "../apig-resign.js"; import { conflictScanEnabled, isDesignBenign, scanClientPlugins, sniffScanClient } from "../thirdparty-scan.js"; import { recordConflict } from "../conflict-watch.js"; @@ -62,8 +62,8 @@ import { evaluateChain, extractChainCarriers } from "../chain-checkpoint.js"; import { BILI_PASSTHROUGH_HEADER, BILI_PLUGIN_BYPASS_HEADER, isLoopbackAddress, normalizeUpstreamOrigin, provenCompletionReserve, reserveOutputHeadroom, resolveOutputHeadroomCap, shouldReserveOutputHeadroom, type WireProtocol } from "../util.js"; import { checkTunnelDestination, tunnelAllowlistFromEnv } from "../tunnel-guard.js"; import { resolveCompatDropFields } from "../compat-drop.js"; -import { DecompressedTooLargeError, decodeRequestBody } from "../content-encoding.js"; -import { autoModeForgeEnabled, dshAuthoredCompactionEnabled, imageRescaleEnabled, limitsDecodedBytes as knobLimitsDecodedBytes, noInjectTool as knobNoInjectTool, rawDumpDir as knobRawDumpDir } from "../knobs.js"; +import { DecompressedTooLargeError, decodeRequestBody, IngestBodyTooLargeError } from "../content-encoding.js"; +import { autoModeForgeEnabled, dshAuthoredCompactionEnabled, imageRescaleEnabled, limitsDecodedBytes as knobLimitsDecodedBytes, limitsIngestBytes as knobLimitsIngestBytes, noInjectTool as knobNoInjectTool, rawDumpDir as knobRawDumpDir } from "../knobs.js"; import { buildForgeResponse, classifierForgeVariant, estimateCjkTokens, isAutoModeClassifier, requestText, summarizeAction } from "../auto-mode-forge.js"; import { extractConversationId, isV1BackgroundSideRequest, isV1InternalRequest, unwrapV1Internal } from "../v1internal.js"; import { anthropicBetaContextWindow, BILI_HOP_HEADER, capRegistryWindowByStandard, expandedContextSuffixWindow, launcherContextWindow, launcherMaxOutput, noteUnresolvedModelSession, UNRESOLVED_MODEL_SESSION_THRESHOLD, windowSourceLogged } from "./context-window.js"; @@ -436,10 +436,35 @@ async function handleRequest( // size here; idempotent — a compressed body already observed // via onBytes is admitted and this no-ops. admission.observe(decoded.body.byteLength); + // #5794 item 2: the slot budget above caps CONCURRENCY of large + // decodes; a SINGLE oversized body still burns the event loop + // for seconds in the JSON.parse + normalization sweep (the + // 51.2 MiB / 29,619-message incident: 5.5–7.2 s per round, + // 54.6% of local CPU, 747 s FATAL → cold caches for every + // session). Any TEXT-dominated parseable body above the ingest + // cost band is already ≥ ~8M tokens — beyond every real model + // window — so refuse it HERE, after decode (which can balloon + // 10:1) but before the sweep that costs the seconds. 0 disables + // the cap (legacy parse-everything). + // + // Image-bearing bodies are exempt on EVERY lane: base64 pixels + // are excluded from the token meter by design ("images billed + // separately, never doom on raw bytes" — #496/#1800/#857), so + // "over-band = over-window" does not hold for them. #3909's + // ~48 MiB image-dense chat requests and #3701's ~38.5 MiB + // WS-screenshot bodies are supported workloads governed by the + // slot budget, the receive cap and each lane's own shed / + // pre-send byte budget. The probe is a cheap substring gate, + // so the text-dump rejection (#5794's shape) stays parse-free. + const ingestCap = protocol === "responses" ? 0 : knobLimitsIngestBytes(); + if (ingestCap > 0 && decoded.body.byteLength > ingestCap + && countImagesInRawBody(protocol, decoded.body) === 0) { + throw new IngestBodyTooLargeError(ingestCap, decoded.body.byteLength); + } bodyBuffer = decoded.body; if (decoded.decoded) delete req.headers["content-encoding"]; } catch (decErr) { - if (decErr instanceof DecompressedTooLargeError || decErr instanceof DecodedRequestBusyError || decodingSignal.aborted) throw decErr; + if (decErr instanceof DecompressedTooLargeError || decErr instanceof IngestBodyTooLargeError || decErr instanceof DecodedRequestBusyError || decodingSignal.aborted) throw decErr; // #619: bili can't decode this content-encoding -> don't 400. Drop // protocol so the request falls to the verbatim passthrough below, // relaying the ORIGINAL still-encoded bytes (the reassignment never @@ -457,10 +482,10 @@ async function handleRequest( res.end(JSON.stringify({ error: { type: "request_body_busy", stage: "decode", message: err.message } })); return; } - if (err instanceof BodyTooLargeError || err instanceof DecompressedTooLargeError) { + if (err instanceof BodyTooLargeError || err instanceof DecompressedTooLargeError || err instanceof IngestBodyTooLargeError) { log("warn", `413: request body exceeds ${err.limit} bytes`); res.writeHead(413, { "content-type": "application/json" }); - res.end(JSON.stringify({ error: { type: "request_too_large", stage: err instanceof BodyTooLargeError ? "receive" : "decode", message: err.message } })); + res.end(JSON.stringify({ error: { type: "request_too_large", stage: err instanceof IngestBodyTooLargeError ? "ingest" : err instanceof BodyTooLargeError ? "receive" : "decode", message: err.message } })); return; } log("warn", `failed to prepare inbound request (${String(err)}) - 400`); diff --git a/src/server/relay.ts b/src/server/relay.ts index d56c3ce816..d3e4e521fb 100644 --- a/src/server/relay.ts +++ b/src/server/relay.ts @@ -33,7 +33,7 @@ import { currentFetchTransport } from "../fetch-transport.js"; import { refForItemBytes } from "../image-rescale.js"; import { countImagesInParsedBody, countImagesInRawBody, upstreamHost } from "../image-tokens.js"; import { log as loggerLog } from "../logger.js"; -import { dumpReqAllowed as knobDumpReqAllowed, limitsWsOutboundBytes, rawDumpDir as knobRawDumpDir } from "../knobs.js"; +import { dumpReqAllowed as knobDumpReqAllowed, limitsMaxMessages, limitsWsOutboundBytes, rawDumpDir as knobRawDumpDir } from "../knobs.js"; import { gcDumpDirIfConfigured } from "../state-gc.js"; import { applyLaneCredential, laneCredential } from "../lane-credentials.js"; import { CREDENTIAL_HEADER_RE, maskHeaderForLog, maskHeadersForLog, maskHostPortForDisplay, maskUrlForLog } from "../log-mask.js"; @@ -69,7 +69,7 @@ import { buildForwardHeaders, connectionNamedHeaders, RESPONSE_ONLY_STRIP_HEADER import { bodyDumpEnabled, logDumpFailure } from "./observability.js"; import { effectiveAbsorbBlock } from "./prepare-responses.js"; import { awaitDrain, bufferToStream, dumpStreamToFile, pipeThrough, readStreamToBuffer } from "./stream-io.js"; -import { buildForwardTarget, makeFoldPointReasoningDroppable, measuredOutboundTokens, outboundPayloadBreakdown, registerRequestAbort, beginStreamKeepalive, effectiveTokenCount, googlePathKind, imageBillingFor, imageTokenCapFor, inboundResignCredential, outboundContextEstimates, repairResponsesAssistantOrdering, resolveUpstream, stripKernelSummaries, usageGradeInputBaseline, DOOMED_PROBE_ABSOLUTE_BYTES, DOOMED_PROBE_BYTES_PER_TOKEN, type Prepared } from "../server.js"; +import { buildForwardTarget, makeFoldPointReasoningDroppable, measuredBreakdownClause, measuredOutbound, outboundPayloadBreakdown, registerRequestAbort, beginStreamKeepalive, effectiveTokenCount, googlePathKind, imageBillingFor, imageTokenCapFor, inboundResignCredential, outboundContextEstimates, repairResponsesAssistantOrdering, resolveUpstream, stripKernelSummaries, usageGradeInputBaseline, DOOMED_PROBE_ABSOLUTE_BYTES, DOOMED_PROBE_BYTES_PER_TOKEN, type Prepared } from "../server.js"; import { isStrictReasoningEcho, modelIdOf } from "../strict-echo.js"; /** Infer the wire protocol from the request path for compat-role rewrites on @@ -740,10 +740,27 @@ async function forwardCaptured( // window sits ABOVE the 1M floor, so floored or not the 1.12M-token // body exceeds it. The byte floor stays as the fallback for // unparseable bodies. No refusal record either way (#3974). - const fuseRawTokens = measuredOutboundTokens(prepared); - if (fuseRawTokens !== undefined && fuseRawTokens > fuseWindow) { + // #5000 item 2: the COUNT dimension — no window, tokenizer, or byte + // floor sees the messages[] array length (the 26,903-message #5000 + // session cleared every size gate and died at the upstream). Same + // per-upstream opt-in as preflight (BILI_MAX_MESSAGES); the fuse is + // the last line for the no-model lane where preflight never ran. + const fuseMaxMessages = limitsMaxMessages(); + if (fuseMaxMessages > 0 && fuseMsgs.length > fuseMaxMessages) { + const countMessage = `message count ${fuseMsgs.length.toLocaleString("en-US")} exceeds the upstream wire limit ${fuseMaxMessages.toLocaleString("en-US")} (COUNT dimension — a hard upstream constraint on the messages[] array that no byte/token estimate expresses, #5000); the send was stopped at the egress fuse instead of being forwarded. Reduce the session history — restart the session or rewind to an earlier point — and retry.`; + log("warn", `413: [${prepared.session.id}] egress window fuse count gate: ${countMessage}`); + if (res.headersSent) emitStreamError(res, prepared.protocol, countMessage, undefined, opts.streamErrorShape); + else { + res.writeHead(413, { "content-type": "application/json" }); + res.end(JSON.stringify({ error: { type: "request_too_large", code: "egress_window_fuse", stage: "forward", message: countMessage, retryable: false } })); + } + return; + } + const fuseMeasured = measuredOutbound(prepared); + const fuseRawTokens = fuseMeasured?.tokens; + if (fuseMeasured !== undefined && fuseRawTokens !== undefined && fuseRawTokens > fuseWindow) { log("warn", `413: [${prepared.session.id}] egress window fuse measured-body profile: ${describeCoreTextDuplication(fuseMsgs)}`); - const fuseMessage = `outbound body measures ~${fuseRawTokens.toLocaleString("en-US")} tokens (system + tools + arguments + messages; CJK-aware count, the #554 guard meter) vs the effective model window ${declaredWindow.toLocaleString("en-US")} (bars never assume below ${MIN_ASSUMED_WINDOW_TOKENS} tokens, #4079) (model=${prepared.model ?? "unknown"}); the body ITSELF exceeds the window, so the send was stopped at the egress fuse instead of being forwarded (#5000/#4543/#4068). Remove or trim the oversized content (e.g. a giant paste or tool result) and retry.`; + const fuseMessage = `outbound body measures ~${fuseRawTokens.toLocaleString("en-US")} tokens (${measuredBreakdownClause(fuseMeasured.breakdown)}; CJK-aware count, the #554 guard meter) vs the effective model window ${declaredWindow.toLocaleString("en-US")} (bars never assume below ${MIN_ASSUMED_WINDOW_TOKENS} tokens, #4079) (model=${prepared.model ?? "unknown"}); the body ITSELF exceeds the window, so the send was stopped at the egress fuse instead of being forwarded (#5000/#4543/#4068). Remove or trim the oversized dimension named above (e.g. a giant paste or tool result in messages[], or tools[] schemas) and retry.`; log("warn", `413: [${prepared.session.id}] egress window fuse: ${fuseMessage}`); if (res.headersSent) emitStreamError(res, prepared.protocol, fuseMessage, undefined, opts.streamErrorShape); else { diff --git a/tests/issue5000-count-gate.test.ts b/tests/issue5000-count-gate.test.ts new file mode 100644 index 0000000000..44d41b854f --- /dev/null +++ b/tests/issue5000-count-gate.test.ts @@ -0,0 +1,207 @@ +// #5000 item 2: the COUNT dimension — a hard upstream wire constraint on the +// messages[] array length that no byte/token estimate can express. Incident +// #5000's 55.2 MiB / 26,903-message session cleared EVERY size gate (core +// text 1.44 MiB, the 2 MiB absolute floor, the egress byte floor) and died at +// the upstream with the count error. This suite pins the per-upstream opt-in +// gate (BILI_MAX_MESSAGES=500 here): over-count bodies answer an actionable +// 413 (restart/rewind) with ZERO upstream calls — including token-fit bodies +// (the gate runs BEFORE preflight's token trigger, which is exactly the +// #5000 case), and under-count bodies forward unchanged. +import assert from "node:assert/strict"; +import http from "node:http"; +import { once } from "node:events"; +import test from "node:test"; +import { mkdtempSync } from "node:fs"; +import { tmpdir } from "node:os"; +import path from "node:path"; + +process.env.NODE_ENV = "test"; +// The gate is per-upstream: 0/unset = disabled (the default — no global +// value exists). node:test can run several files in ONE process (the CI test +// lanes do), so the limit is armed per-test and RESTORED in finally — a +// module-scope pin would leak into sibling suites' sessions. +function armCountLimit(): () => void { + const prev = process.env.BILI_MAX_MESSAGES; + process.env.BILI_MAX_MESSAGES = String(COUNT_LIMIT); + return () => { + if (prev === undefined) delete process.env.BILI_MAX_MESSAGES; + else process.env.BILI_MAX_MESSAGES = prev; + }; +} + +import { defaultConfig } from "acp-kernel"; +import { startServer } from "../src/server.ts"; +import type { ProxyOptions } from "../src/config.ts"; +import { SessionStore, _setStoreForTest } from "../src/persist.ts"; +import { _setForTest as setRegistryForTest } from "../src/registry.ts"; +import { rmrf } from "./tmp-rm.ts"; + +const WINDOW = 20_000; +const COUNT_LIMIT = 500; + +type Call = { bytes: number }; + +function chatSse(text: string): string { + const chunk = (delta: Record, finish: string | null, extra?: Record): string => + `data: ${JSON.stringify({ id: "chatcmpl-1", object: "chat.completion.chunk", created: 1, model: "gpt-test", choices: [{ index: 0, delta, finish_reason: finish }], ...extra })}\n\n`; + return chunk({ role: "assistant" }, null) + chunk({ content: text }, null) + chunk({}, "stop", { usage: { prompt_tokens: 100, completion_tokens: 3 } }) + "data: [DONE]\n\n"; +} + +function tinyMessages(count: number): Array<{ role: string; content: string }> { + // ~4 chars of CONTENT per message, but the rebuilt body prepends an + // ~69-byte mNNNNN ref marker to every message (the m-ref + // ledger the compress tooling relies on), so each message costs ~20.8 + // tokens on the wire. 600 msgs ≈ 12.5K tokens < the 15K budget → the + // body is TOKEN-FIT, yet its count (600) exceeds the limit (500): the + // only dimension that can reject it is COUNT. That is the real #5000 + // shape (26,903 ref-marked messages that cleared every size gate). + const messages: Array<{ role: string; content: string }> = []; + for (let i = 0; i < count; i++) { + messages.push({ role: i % 2 === 0 ? "user" : "assistant", content: "hi i" }); + } + return messages; +} + +function makeUpstream(): { server: http.Server; calls: Call[] } { + const calls: Call[] = []; + const server = http.createServer((req, res) => { + const chunks: Buffer[] = []; + req.on("data", (c: Buffer) => chunks.push(c)); + req.on("end", () => { + calls.push({ bytes: Buffer.concat(chunks).length }); + res.writeHead(200, { "content-type": "text/event-stream" }); + res.end(chatSse("forwarded answer")); + }); + }); + return { server, calls }; +} + +function proxyOptions(routes: Record): ProxyOptions { + return { + port: 0, + host: "127.0.0.1", + upstream: "http://127.0.0.1", + routes: routes as ProxyOptions["routes"], + modelContextLimit: WINDOW, + kernelConfig: defaultConfig(WINDOW, { + preserveRecentMessages: 2, + preserveRecentTokens: 2000, + compress: { minCompressRange: 1000, maxSummaryLength: 20000, minSummaryLength: 50 }, + }), + compress: { injectTool: true, injectNudge: true }, + sessionHeader: "x-acp-session", + log: false, + debug: false, + passthrough: false, + autoUpdate: false, + compat: { roles: {} }, + streamErrorShape: "protocol", + passthroughSource: null, + autoRestartOnUpdate: false, + updateTag: "latest", + advisoryCheck: false, + releaseNotesCheck: false, + promptCache: { routing: "auto" }, + mitm: { enabled: false, domains: [] }, + } as ProxyOptions; +} + +type Ctx = { url: string; headers: Record }; + +async function setup(): Promise<{ ctx: Ctx; calls: Call[]; close: () => void }> { + _setStoreForTest(new SessionStore({ enabled: false })); + setRegistryForTest({}); + const { server: upstream, calls } = makeUpstream(); + upstream.listen(0, "127.0.0.1"); + await once(upstream, "listening"); + const upstreamPort = (upstream.address() as { port: number }).port; + const proxy = await startServer(proxyOptions({ [`http://127.0.0.1:${upstreamPort}`]: { models: { "gpt-test": { context: WINDOW } } } })); + await once(proxy, "listening"); + const proxyPort = (proxy.address() as { port: number }).port; + return { + ctx: { + url: `http://127.0.0.1:${proxyPort}/bili/http://127.0.0.1:${upstreamPort}/chat/completions`, + headers: { "content-type": "application/json", "x-acp-session": "s-count-ingest" }, + }, + calls, + close: () => { + proxy.close(); + proxy.closeAllConnections?.(); + upstream.close(); + upstream.closeAllConnections?.(); + }, + }; +} + +const post = (ctx: Ctx, messages: Array<{ role: string; content: string }>, withModel: boolean) => + fetch(ctx.url, { + method: "POST", + headers: { ...ctx.headers }, + body: JSON.stringify({ ...(withModel ? { model: "gpt-test" } : {}), stream: true, messages }), + }); + +test(`e2e #5000: a TOKEN-FIT body over the upstream's count limit is answered 413 with the COUNT dimension — zero upstream calls (BILI_MAX_MESSAGES=${COUNT_LIMIT})`, async () => { + const root = mkdtempSync(path.join(tmpdir(), "bili-5000a-")); + const disarm = armCountLimit(); + const { ctx, calls, close } = await setup(); + try { + // 600 tiny messages: ~12.5K tokens < 15K budget (token-fit — the + // token trigger never fires, so preflight would early-return and the + // body would sail to a guaranteed count-400 upstream). The count gate + // runs BEFORE that trigger, so it is the only gate that can see this + // shape. + const r = await post(ctx, tinyMessages(600), true); + const body = await r.text(); + const parsed = JSON.parse(body) as { error?: { type?: string; code?: string; retryable?: boolean; message?: string } }; + + assert.equal(r.status, 413, `over-count body must fail fast (got ${r.status}): ${body.slice(0, 300)}`); + assert.equal(parsed.error?.code, "preflight_compress_failed", "the preflight fail-fast house shape"); + assert.equal(parsed.error?.retryable, false, "retrying re-sends the same 600-message history (#2847 semantics)"); + assert.match(parsed.error?.message ?? "", /message count 600 exceeds the upstream wire limit 500/, "the COUNT math is in the operator-facing message"); + assert.match(parsed.error?.message ?? "", /COUNT dimension/, "the dimension names itself — it is not a byte/token judgment"); + assert.match(parsed.error?.message ?? "", /restart the session or rewind/, "the actionable remedy is present (folding cannot close the gap in one preflight)"); + // THE behavior under test: the doomed payload NEVER reaches the upstream + // (before the gate this exact shape cleared every size gate — #5000). + assert.equal(calls.length, 0, `upstream must see zero calls (got ${JSON.stringify(calls)})`); + } finally { + disarm(); + await close(); + rmrf(root); + } +}); + +test("e2e #5000: the SAME over-count body without a model (preflight's !model early return) still gets the count 413 — the gate precedes the model check", async () => { + const root = mkdtempSync(path.join(tmpdir(), "bili-5000b-")); + const disarm = armCountLimit(); + const { ctx, calls, close } = await setup(); + try { + const r = await post(ctx, tinyMessages(600), false); + const body = await r.text(); + const parsed = JSON.parse(body) as { error?: { type?: string; retryable?: boolean; message?: string } }; + + assert.equal(r.status, 413, `no-model over-count body must fail fast (got ${r.status}): ${body.slice(0, 300)}`); + assert.equal(parsed.error?.retryable, false, "non-retryable — retrying re-sends the same history"); + assert.match(parsed.error?.message ?? "", /message count 600 exceeds the upstream wire limit 500/, "the COUNT math is present"); + assert.equal(calls.length, 0, `upstream must see zero calls (got ${JSON.stringify(calls)})`); + } finally { + disarm(); + await close(); + rmrf(root); + } +}); + +test("e2e #5000 control: under-count bodies forward unchanged when the gate is armed (the gate must not false-block)", async () => { + const root = mkdtempSync(path.join(tmpdir(), "bili-5000c-")); + const disarm = armCountLimit(); + const { ctx, calls, close } = await setup(); + try { + const under = await post(ctx, tinyMessages(400), true); + const ub = await under.text(); + assert.equal(under.status, 200, `400 messages < the 500 limit must forward (got ${under.status}): ${ub.slice(0, 300)}`); + assert.equal(calls.length, 1, "the under-count body reached the upstream"); + } finally { + disarm(); + await close(); + rmrf(root); + } +}); diff --git a/tests/issue5794-ingest-cap.test.ts b/tests/issue5794-ingest-cap.test.ts new file mode 100644 index 0000000000..e9a5c6c266 --- /dev/null +++ b/tests/issue5794-ingest-cap.test.ts @@ -0,0 +1,206 @@ +// #5794 item 2: the INGEST cost band. The #5794 session sent 51.2 MiB (29,619 +// messages) every round; the slot budget (32 MiB admission, #5932) capped +// CONCURRENCY but a single body still burned the event loop for 5.5–7.2 s per +// round in the JSON.parse + normalization sweep (54.6% of local CPU → 747 s +// FATAL → every session's cache went cold). Any parseable body over the band +// (default 32 MiB) is already ≥ ~8M tokens — beyond every real model window — +// so it is refused with an actionable 413 (stage ingest) BEFORE the sweep. +// This suite pins both directions with a small band (1 MiB) so the over-cap +// body stays a few MiB: over → 413 with the dimension + remedy, under → 200. +import assert from "node:assert/strict"; +import http from "node:http"; +import { once } from "node:events"; +import test from "node:test"; +import { mkdtempSync } from "node:fs"; +import { tmpdir } from "node:os"; +import path from "node:path"; + +process.env.NODE_ENV = "test"; +// 1 MiB test band. node:test can run several files in ONE process (the CI +// test lanes do), so the band is armed per-test and RESTORED in finally. +const TEST_BAND = 1024 * 1024; +function armIngestBand(): () => void { + const prev = process.env.BILI_INGEST_BODY_BYTES; + process.env.BILI_INGEST_BODY_BYTES = String(TEST_BAND); + return () => { + if (prev === undefined) delete process.env.BILI_INGEST_BODY_BYTES; + else process.env.BILI_INGEST_BODY_BYTES = prev; + }; +} + +import { defaultConfig } from "acp-kernel"; +import { startServer } from "../src/server.ts"; +import type { ProxyOptions } from "../src/config.ts"; +import { SessionStore, _setStoreForTest } from "../src/persist.ts"; +import { _setForTest as setRegistryForTest } from "../src/registry.ts"; +import { rmrf } from "./tmp-rm.ts"; + +const WINDOW = 20_000; + +type Call = { bytes: number }; + +function chatSse(text: string): string { + const chunk = (delta: Record, finish: string | null, extra?: Record): string => + `data: ${JSON.stringify({ id: "chatcmpl-1", object: "chat.completion.chunk", created: 1, model: "gpt-test", choices: [{ index: 0, delta, finish_reason: finish }], ...extra })}\n\n`; + return chunk({ role: "assistant" }, null) + chunk({ content: text }, null) + chunk({}, "stop", { usage: { prompt_tokens: 100, completion_tokens: 3 } }) + "data: [DONE]\n\n"; +} + +function makeUpstream(): { server: http.Server; calls: Call[] } { + const calls: Call[] = []; + const server = http.createServer((req, res) => { + const chunks: Buffer[] = []; + req.on("data", (c: Buffer) => chunks.push(c)); + req.on("end", () => { + calls.push({ bytes: Buffer.concat(chunks).length }); + res.writeHead(200, { "content-type": "text/event-stream" }); + res.end(chatSse("forwarded answer")); + }); + }); + return { server, calls }; +} + +function proxyOptions(routes: Record, window = WINDOW): ProxyOptions { + return { + port: 0, + host: "127.0.0.1", + upstream: "http://127.0.0.1", + routes: routes as ProxyOptions["routes"], + modelContextLimit: window, + kernelConfig: defaultConfig(window, { + preserveRecentMessages: 2, + preserveRecentTokens: 2000, + compress: { minCompressRange: 1000, maxSummaryLength: 20000, minSummaryLength: 50 }, + }), + compress: { injectTool: true, injectNudge: true }, + sessionHeader: "x-acp-session", + log: false, + debug: false, + passthrough: false, + autoUpdate: false, + compat: { roles: {} }, + streamErrorShape: "protocol", + passthroughSource: null, + autoRestartOnUpdate: false, + updateTag: "latest", + advisoryCheck: false, + releaseNotesCheck: false, + promptCache: { routing: "auto" }, + mitm: { enabled: false, domains: [] }, + } as ProxyOptions; +} + +type Ctx = { url: string; headers: Record }; + +async function setup(window = WINDOW): Promise<{ ctx: Ctx; calls: Call[]; close: () => void }> { + _setStoreForTest(new SessionStore({ enabled: false })); + setRegistryForTest({}); + const { server: upstream, calls } = makeUpstream(); + upstream.listen(0, "127.0.0.1"); + await once(upstream, "listening"); + const upstreamPort = (upstream.address() as { port: number }).port; + const proxy = await startServer(proxyOptions({ [`http://127.0.0.1:${upstreamPort}`]: { models: { "gpt-test": { context: window } } } }, window)); + await once(proxy, "listening"); + const proxyPort = (proxy.address() as { port: number }).port; + return { + ctx: { + url: `http://127.0.0.1:${proxyPort}/bili/http://127.0.0.1:${upstreamPort}/chat/completions`, + headers: { "content-type": "application/json", "x-acp-session": "s-count-ingest" }, + }, + calls, + close: () => { + proxy.close(); + proxy.closeAllConnections?.(); + upstream.close(); + upstream.closeAllConnections?.(); + }, + }; +} + +const postMessages = (ctx: Ctx, content: string) => + fetch(ctx.url, { + method: "POST", + headers: { ...ctx.headers }, + body: JSON.stringify({ model: "gpt-test", stream: true, messages: [{ role: "user", content }] }), + }); + +test("e2e #5794: a decoded body over the ingest cost band is refused 413 (stage ingest) BEFORE the parse sweep — zero upstream calls", async () => { + const root = mkdtempSync(path.join(tmpdir(), "bili-5794a-")); + const disarm = armIngestBand(); + const { ctx, calls, close } = await setup(); + try { + // ~3 MiB of content in ONE message: over the 1 MiB test band, far + // under the 5 GiB receive cap and the 32 MiB slot band (so the + // admission budget is not the thing answering), and identity-encoded + // (the #5794 shape: plain JSON, no content-encoding). + const r = await postMessages(ctx, "x".repeat(3 * 1024 * 1024)); + const body = await r.text(); + const parsed = JSON.parse(body) as { error?: { type?: string; stage?: string; message?: string } }; + + assert.equal(r.status, 413, `over-band body must be refused (got ${r.status}): ${body.slice(0, 300)}`); + assert.equal(parsed.error?.type, "request_too_large", "the structured too-large family"); + assert.equal(parsed.error?.stage, "ingest", "the STAGE names the ingest band (distinct from receive/decode caps)"); + assert.match(parsed.error?.message ?? "", /exceeds the ingestion cost band/, "the band is named for the operator"); + assert.match(parsed.error?.message ?? "", /restart the session or rewind/, "the actionable remedy is present"); + // THE behavior under test: the body never enters the parse sweep and + // never reaches the upstream (on master this body cost 5.5–7.2 s of + // event loop per round before any window logic could even look at it). + assert.equal(calls.length, 0, `upstream must see zero calls (got ${JSON.stringify(calls)})`); + } finally { + disarm(); + await close(); + rmrf(root); + } +}); + +test("e2e #5794 control: a body under the ingest band is unaffected (the band must not false-block)", async () => { + const root = mkdtempSync(path.join(tmpdir(), "bili-5794b-")); + const disarm = armIngestBand(); + const { ctx, calls, close } = await setup(); + try { + // 30 KB ≈ ~7.5K tokens: under the 1 MiB band AND under the 20K window + // (so preflight's token trigger stays quiet too) — a plain normal turn. + const under = await postMessages(ctx, "x".repeat(30 * 1024)); + assert.equal(under.status, 200, `30 KiB < the 1 MiB band must be accepted (got ${under.status})`); + await under.text(); + assert.equal(calls.length, 1, "the under-band body reached the upstream"); + } finally { + disarm(); + await close(); + rmrf(root); + } +}); + +test("e2e #5794 image doctrine: an over-band body carrying images is NOT refused at ingest (pixels never doom on raw bytes)", async () => { + const root = mkdtempSync(path.join(tmpdir(), "bili-5794c-")); + const disarm = armIngestBand(); + // b64 pixels bill at ~1/4 token per char, so a ~1.6 MiB image body is + // ~400K tokens in token-space — over the file's 20K window but far under + // the 1M window this test arms. The point is the INGEST stage, not the + // token meter: an over-band IMAGE body must flow through to the upstream + // (governed by the slot budget / receive cap / lane shed), while an + // over-band TEXT body is refused before the sweep. + const { ctx, calls, close } = await setup(1_000_000); + try { + const b64 = Buffer.alloc(600 * 1024, 1).toString("base64"); // ~800 KB each + const r = await fetch(ctx.url, { + method: "POST", + headers: { ...ctx.headers }, + body: JSON.stringify({ + model: "gpt-test", + stream: true, + messages: [{ role: "user", content: [ + { type: "text", text: "what is in these?" }, + { type: "image_url", image_url: { url: `data:image/png;base64,${b64}` } }, + { type: "image_url", image_url: { url: `data:image/png;base64,${b64}` } }, + ] }], + }), + }); + const body = await r.text(); + assert.equal(r.status, 200, `over-band IMAGE body must not be refused at ingest (got ${r.status}): ${body.slice(0, 300)}`); + assert.equal(calls.length, 1, "the image body reached the upstream — image mass is governed by the image lanes, not the byte band"); + } finally { + disarm(); + await close(); + rmrf(root); + } +}); diff --git a/tests/side-request-isolation.test.ts b/tests/side-request-isolation.test.ts index 932b9b97d6..c9b0813bf3 100644 --- a/tests/side-request-isolation.test.ts +++ b/tests/side-request-isolation.test.ts @@ -6,7 +6,7 @@ import { defaultConfig, createInitialState, defaultCountTokens } from "acp-kerne import { startServer, isSideRequest, outputBudgetField, restoreOutputBudget, sideRequestGuard, resolveKnownOutputCeiling, _resetNoOutputCeilingWarningsForTest, _resetDshMidBandWarningsForTest, OUTPUT_RESTORE_ABSOLUTE_FLOOR } from "../src/server.ts"; import type { ProxyOptions } from "../src/config.ts"; import { recordPluginRuntimeInfo, _resetPluginStateForTest } from "../src/plugin.ts"; -import { estimateRawBodyTokens } from "../src/preflight.ts"; +import { estimateRawBodyTokens, estimateRawBodyBreakdown } from "../src/preflight.ts"; import { inspectContextOverflow } from "../src/util.ts"; import { SessionStore, _setStoreForTest } from "../src/persist.ts"; import { mkdtempSync } from "node:fs"; @@ -554,6 +554,30 @@ test("estimateRawBodyTokens: counts string leaves, skips binary-carrying keys (# assert.ok(estimateRawBodyTokens({ content: "汉".repeat(100) }) >= 100, "CJK counted per-char"); }); +test("estimateRawBodyBreakdown: splits the same total on the top-level field (#5000 item 3)", () => { + const toolsMass = 120 * defaultCountTokens("d".repeat(40_000)); + const body = { + system: "s".repeat(1_000), + tools: Array.from({ length: 120 }, () => ({ type: "function", function: { name: "t", description: "d".repeat(40_000) } })), + messages: [{ role: "user", content: "m".repeat(1_000) }], + }; + const b = estimateRawBodyBreakdown(body); + assert.equal(b.total, estimateRawBodyTokens(body), "total equals the plain meter exactly (same walk, same exclusions)"); + assert.ok(b.tools >= toolsMass, "tools[] mass lands in the tools bucket"); + assert.ok(b.messages >= defaultCountTokens("m".repeat(1_000)), "message text lands in the messages bucket"); + assert.ok(b.system >= defaultCountTokens("s".repeat(1_000)), "system text lands in the system bucket"); + assert.equal(b.total, b.messages + b.tools + b.system + b.other, "buckets are exhaustive"); + // the #5000 shape: a body whose mass rides tools[] — the doom message must + // be able to name THAT dimension, not "trim the oversized content". + assert.ok(b.tools > b.messages + b.system + b.other, "tools-dominant body reports tools-dominant"); + // image payloads still excluded from every bucket (#5932). + const img = estimateRawBodyBreakdown({ image_url: `data:image/png;base64,${"A".repeat(50_000)}` }); + assert.equal(img.total, 0, "image_url stays excluded in every bucket"); + // non-object bodies bucket into other (or nothing) without throwing. + assert.equal(estimateRawBodyBreakdown(null).total, 0, "null body"); + assert.equal(estimateRawBodyBreakdown("plain string").total, defaultCountTokens("plain string"), "bare string counts as other"); +}); + test("inspectContextOverflow: exceed_context_size_error pattern + (A / B > W) window parse (#554)", () => { const llama = JSON.stringify({ error: { message: "exceed_context_size_error (198,277 / 198,661 > 150,528)" } }); const hit = inspectContextOverflow(400, llama); diff --git a/website/config-reference/process.yaml b/website/config-reference/process.yaml index d13d8420d8..29e8811f1e 100644 --- a/website/config-reference/process.yaml +++ b/website/config-reference/process.yaml @@ -187,6 +187,20 @@ entries: group: process desc_en: "Soft byte budget for the plain/proxy/pi-native HTTP relay lane (#3909): when the final serialized request exceeds it, older images are shed from the OUTBOUND COPY first (archived as placeholder notes — originals stay in session storage, recoverable via read_original_image / decompress on tool-armed lanes — codex-style lanes that reject injected tools keep the size-only annotation), then whatever remains is boundedly downscaled through the jimp maxDim ladder. Closes the gap image-dense sessions leave open: token-space folding decides in tokens while the upstream rejects in bytes / pixel-decode CPU (Cloudflare Worker 503), so a ~48 MiB request 503s forever with every retry resending the same pixels. Applies only when no managed transport is installed (the spawned relay); the WS lane uses network.wsOutboundBytes and managed HTTP transports (inline-engine / dsh-inline) are never shed here (#3701 Finding 1). Set 0 to disable (previous send-as-is); the hard refusal cap stays network.outboundBodyBytes." desc_zh: "面向普通/代理/pi 原生 HTTP 中继通道的软字节预算(#3909):当最终序列化的请求超出该预算时,优先从出站副本中裁减较早的图片(归档为占位符备注——原始文件仍保留在会话存储中,可通过 read_original_image / decompress 恢复——仅限注册了 bili 工具的通道;拒绝注入工具的 codex 类通道只有尺寸备注),随后对剩余内容通过 jimp maxDim 阶梯进行有界降采样。它补上图片密集会话留下的缺口:token 空间折叠按 token 决策,而上游按字节/像素解码 CPU 拒收(Cloudflare Worker 503),于是 ~48 MiB 的请求会永远 503、每次重试都重发同样的像素。仅在未安装受管传输(即 spawn 出来的中继)时生效;WebSocket 通道用 network.wsOutboundBytes,受管 HTTP 传输(inline-engine / dsh-inline)在此从不裁减(#3701 Finding 1)。设为 0 可禁用(恢复此前的原样发送);硬性拒绝上限仍为 network.outboundBodyBytes。" + - key: network.ingestBodyBytes + type: number + default: "33554432" + env: BILI_INGEST_BODY_BYTES + group: process + desc_en: "Cost band (decoded bytes) for the JSON.parse + normalization sweep AFTER decode (#5794 item 2). A TEXT-DOMINATED body above it is refused with an actionable 413 (stage ingest) before the sweep, because any parseable text body over 32 MiB is already ~8M tokens — beyond every real model window — and burning seconds of event-loop time to parse it (the 51.2 MiB / 29,619-message incident: 5.5–7.2 s per round, 54.6% of local CPU, 747 s FATAL) cannot end in success. Image-bearing bodies are exempt on every lane: base64 pixels are excluded from the token meter by design (images billed separately, never doomed on raw bytes — #496/#1800/#857), so over-band image mass stays governed by the admission slots, the receive cap and each lane's own shed / pre-send byte budget (#3909 ~48 MiB image-dense sessions, #3701 WS screenshots; the Responses lane is exempt outright regardless). Distinct from the admission SLOTS (network.receiveBodyBytes threshold): the slots cap CONCURRENCY of large decodes, this cap bounds a single body. Set 0 to disable (legacy parse-everything)." + desc_zh: "解码之后进入 JSON.parse+归一化清扫的成本带(解码后字节,#5794 第 2 项)。超限的**文本为主**的体在清扫之前以可操作的 413(stage ingest)拒收:任何可解析的 32 MiB+ 文本体都 ≥ ~8M token、超过一切真实模型窗口,为它烧掉数秒事件循环时间没有意义(51.2 MiB / 29,619 条消息现场:每轮 5.5–7.2s、占本机 CPU 54.6%、747s FATAL)。带图片的体在所有车道豁免:base64 像素按设计不进 token 计量(图片单独计费、绝不在原始字节上判死——#496/#1800/#857),超带的图片体量仍由准入槽位、收件上限与各车道自身的裁减/预发送字节预算治理(#3909 ~48 MiB 图片密集会话、#3701 WS 截图;Responses 车道无论如何整体豁免)。与大解码准入槽位(network.receiveBodyBytes 阈值)不同:槽位限的是大解码的并发数,本上限限的是单个体的成本。设为 0 可禁用(恢复此前的全解析)。" + - key: network.maxMessages + type: number + default: "0" + env: BILI_MAX_MESSAGES + group: process + desc_en: "Per-upstream hard cap on the messages[] array length (#5000 item 2) — the COUNT dimension: a wire constraint no byte/token estimate expresses (the 26,903-message / 55.2 MiB #5000 session cleared every size gate and died at the upstream with the count error). Per-UPSTREAM on purpose: there is no universal value, so the default is 0 = disabled; set it to your upstream's known limit and over-count bodies get an actionable 413 (restart the session or rewind) instead of a guaranteed upstream 400. Count only — it never folds. Enforced in preflight (before the token trigger) and at the egress fuse for the no-model lane." + desc_zh: "单上游 messages[] 数组长度硬限(#5000 第 2 项)——COUNT 维度:任何字节/token 估算都表达不了的线路约束(#5000 的 26,903 条消息 / 55.2 MiB 会话清掉所有尺寸闸后仍在上游以条数错误死掉)。刻意按上游设置:没有通用值,默认 0 = 关闭;把它设为你所连上游的已知上限后,超条数的体会得到可操作的 413(重启会话或 rewind)而不是一次必然的上游 400。只判条数、从不折叠。在 preflight(token 触发器之前)与无模型车道的出站熔断处强制执行。" - key: network.caFile type: string default: "unset"