Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions CONFIGURATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -161,7 +161,7 @@ A shallow key (`https://open.bigmodel.cn`) matches every path on that host. A de
- **Type:** `Record<string, { context?: number; output?: number; compress?: CompressSettings }>`
- **Default:** *(none)*
- **Status:** ACTIVE
- **Description:** Maps a model name to its context-window declaration. The LLM `/models` endpoint does **not** return context windows (verified across OpenAI, Anthropic, zhipu, comfly), so the proxy cannot discover them at runtime. `context` is the model's context window in tokens; `output` is the max output size.
- **Description:** Maps a model name to its context-window declaration. The LLM `/models` endpoint does **not** return context windows (verified across OpenAI, Anthropic, zhipu, comfly), so the proxy cannot discover them at runtime. `context` is the model's context window in tokens; `output` is the max output size and serves as the output-headroom fallback when a request carries no output-budget field at all (see [`outputHeadroomMaxPct`](#outputheadroommaxpct)).

**Resolution order (first match wins):** (1) per-request sources — the client's `anthropic-beta` larger-context negotiation, a cooperative plugin's report, and the launcher's per-model windows; (2) this per-model `context` declaration; (3) the **warm** models.dev registry cache, when the model is listed (relay/private hosts match the bare model name against the registry's provider-prefixed entries); (4) the built-in context table. So this per-model `context` declaration **outranks the registry** — set it to the window your relay/private deployment actually serves, and it wins even when models.dev lists a different (usually larger) window for the model. `compress.modelContextLimit` remains the highest-priority source (always wins) when you want to pin the window across every route. Each model entry may also carry a per-model `compress` block (see [Compression Tuning](#compression-tuning)).

Expand Down Expand Up @@ -253,7 +253,7 @@ For each request, the proxy resolves the settings by longest-URL-prefix match (t
- **Type:** `number | string`
- **Default:** `0.25`
- **Status:** ACTIVE
- **Description:** Cap on the output-headroom reservation, as a fraction of the context window: reserved amount = `min(max_tokens, pct × window)`. The reservation keeps the engine's nudge/truncate bands below `window − reserved`, so long replies can't push "input + output" past the window — it applies to APIs that count output against the window (Anthropic Messages is exempt: its input limit is enforced independently of `max_tokens`, so it is excluded). Without a cap, models whose registered max output takes a large share of the window (e.g. `maxTokens` 131072 on a 262144 window) lose most of their input budget and the 75% force-compress threshold fires at about a third of the full window. The `0.25` default bounds that loss while still guaranteeing no overflow at the 95% emergency threshold for any single-turn reply up to 25% of the window; longer replies overflow once and are recovered by the next turn's overflow self-heal. Note the cap only relaxes oversized reservations: when `max_tokens` is already ≤ `pct × window`, the reservation stays the full `max_tokens` (byte-identical to the legacy behavior). Accepts a ratio (`0.25`) or percent string (`"25%"`); set `0` to disable the reservation entirely; `>= 1` restores the legacy full-capability reservation (input + a full-budget reply always fits — what strict backends like SGLang/vLLM enforce). Negative or unparseable values reject the whole `compress` block. Example: 262144-token window, `max_tokens = 131072` → default `0.25` reserves 65536 → effective window 196608 (legacy full reservation: 131072); `max_tokens = 65536` → reserves 65536 → 196608 unchanged (65536 ≤ 25% of the window). Aligned with billion-context-pi (`#207`) via #896.
- **Description:** Cap on the output-headroom reservation, as a fraction of the context window: reserved amount = `min(max_tokens, pct × window)`. **Budget source:** when the request carries no output-budget field at all (`max_tokens` / `max_completion_tokens` / `max_output_tokens`) — e.g. Codex's native Responses path sends no `max_output_tokens` (#924) — the proxy falls back to the model's declared max output: the per-route `providers[url].models[model].output` configuration first, then the models.dev registry output ceiling (bundled snapshot as offline floor); if neither exists, no reservation is made. The same cap applies to the fallback value. The reservation keeps the engine's nudge/truncate bands below `window − reserved`, so long replies can't push "input + output" past the window — it applies to APIs that count output against the window (Anthropic Messages is exempt: its input limit is enforced independently of `max_tokens`, so it is excluded). Without a cap, models whose registered max output takes a large share of the window (e.g. `maxTokens` 131072 on a 262144 window) lose most of their input budget and the 75% force-compress threshold fires at about a third of the full window. The `0.25` default bounds that loss while still guaranteeing no overflow at the 95% emergency threshold for any single-turn reply up to 25% of the window; longer replies overflow once and are recovered by the next turn's overflow self-heal. Note the cap only relaxes oversized reservations: when `max_tokens` is already ≤ `pct × window`, the reservation stays the full `max_tokens` (byte-identical to the legacy behavior). Accepts a ratio (`0.25`) or percent string (`"25%"`); set `0` to disable the reservation entirely; `>= 1` restores the legacy full-capability reservation (input + a full-budget reply always fits — what strict backends like SGLang/vLLM enforce). Negative or unparseable values reject the whole `compress` block. Example: 262144-token window, `max_tokens = 131072` → default `0.25` reserves 65536 → effective window 196608 (legacy full reservation: 131072); `max_tokens = 65536` → reserves 65536 → 196608 unchanged (65536 ≤ 25% of the window). Aligned with billion-context-pi (`#207`) via #896.

#### `maxContextLimit`

Expand Down
4 changes: 2 additions & 2 deletions CONFIGURATION.zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -161,7 +161,7 @@
- **类型:** `Record<string, { context?: number; output?: number; compress?: CompressSettings }>`
- **默认值:** *(无)*
- **状态:** ACTIVE
- **说明:** 将模型名映射到其上下文窗口声明。LLM 的 `/models` 端点**不会**返回上下文窗口大小(已在 OpenAI、Anthropic、zhipu、comfly 上验证),因此代理无法在运行时发现它们 —— 你必须在此声明。`context` 是模型的上下文窗口(以 token 为单位);`output` 是最大输出大小。当模型未声明时,代理回退到内置上下文表或 models.dev 注册表。每个模型条目还可以携带按模型的 `compress` 块(见[压缩调优](#压缩调优))。
- **说明:** 将模型名映射到其上下文窗口声明。LLM 的 `/models` 端点**不会**返回上下文窗口大小(已在 OpenAI、Anthropic、zhipu、comfly 上验证),因此代理无法在运行时发现它们 —— 你必须在此声明。`context` 是模型的上下文窗口(以 token 为单位);`output` 是最大输出大小,在请求完全不携带输出预算字段时作为 output headroom 预留的回退值(见 [`outputHeadroomMaxPct`](#outputheadroommaxpct))。当模型未声明时,代理回退到内置上下文表或 models.dev 注册表。每个模型条目还可以携带按模型的 `compress` 块(见[压缩调优](#压缩调优))。

内置上下文表是随每个版本发布的静态数据,可能过期 —— 例如 DeepSeek 的规范请求 id `deepseek-flash` 在 models.dev 上没有以该名列出(其窗口列在 `deepseek-v4-flash` 名下),因此只有兜底表能回答它(#852)。日志会为每个模型记录一次胜出来源(`[window] ... fallback=true` 表示值来自内置表)。若解析出的窗口不对,按上文声明 `models.<name>.context`(它优先于注册表和内置表),或固定 `compress.modelContextLimit`;注意 provider 键必须带流量的 scheme(MITM 登录态客户端流量用 `mitm://<host>`,`/bili/` 流量用 `https://<host>`)。

Expand Down Expand Up @@ -251,7 +251,7 @@
- **类型:** `number | string`
- **默认值:** `0.25`
- **状态:** ACTIVE
- **说明:** 输出预留(output headroom)的上限,以上下文窗口的比例为单位:预留量 = `min(max_tokens, pct × window)`。该预留让引擎的 nudge/truncate 档位位于 `window − 预留量` 之下,防止长回复把「输入+输出」推进窗口之外 —— 适用于把输出计入窗口的 API(Anthropic Messages 豁免:其 input limit 独立于 `max_tokens` 执行,故排除在外)。不设上限时,注册最大输出占窗口比例大的模型(如 262144 窗口上 maxTokens 131072)会失去大半输入预算,75% 强制压缩阈值会在约三分之一的完整窗口处就触发。默认 0.25 在控制损失的同时保证只要单轮回复不超过窗口的 25%,就不会在 95% 紧急阈值下溢出;更长的回复会溢出一次,由下一轮的 overflow self-heal 恢复。注意该上限只放宽过大的预留:当 `max_tokens` 本身 ≤ `pct × window` 时,预留仍是完整的 `max_tokens`(与旧行为逐字节一致)。接受比例(`0.25`)或百分比字符串(`"25%"`);设 `0` 完全禁用预留;`>= 1` 恢复旧的完整预留行为(input + 用满预算的响应总能放进窗口 —— SGLang/vLLM 等严格后端的要求)。负数或无法解析的值会拒绝整个 `compress` 块。示例:窗口 262144 token、`max_tokens = 131072` → 默认 `0.25` 预留 65536 → 有效窗口 196608(旧完整预留:131072);`max_tokens = 65536` → 预留 65536 → 196608 不变(65536 ≤ 窗口的 25%)。与 billion-context-pi(`#207`)对齐,见 #896。
- **说明:** 输出预留(output headroom)的上限,以上下文窗口的比例为单位:预留量 = `min(max_tokens, pct × window)`。**预算来源:** 当请求完全不带输出预算字段(`max_tokens` / `max_completion_tokens` / `max_output_tokens`)时 —— 例如 Codex native Responses 路径不发送 `max_output_tokens`(#924)—— 代理回退到模型声明的最大输出:先取 per-route 配置 `providers[url].models[model].output`,再取 models.dev registry 的 output ceiling(bundled snapshot 离线兜底);都拿不到则不预留。同样的 cap 也作用于回退值。该预留让引擎的 nudge/truncate 档位位于 `window − 预留量` 之下,防止长回复把「输入+输出」推进窗口之外 —— 适用于把输出计入窗口的 API(Anthropic Messages 豁免:其 input limit 独立于 `max_tokens` 执行,故排除在外)。不设上限时,注册最大输出占窗口比例大的模型(如 262144 窗口上 maxTokens 131072)会失去大半输入预算,75% 强制压缩阈值会在约三分之一的完整窗口处就触发。默认 0.25 在控制损失的同时保证只要单轮回复不超过窗口的 25%,就不会在 95% 紧急阈值下溢出;更长的回复会溢出一次,由下一轮的 overflow self-heal 恢复。注意该上限只放宽过大的预留:当 `max_tokens` 本身 ≤ `pct × window` 时,预留仍是完整的 `max_tokens`(与旧行为逐字节一致)。接受比例(`0.25`)或百分比字符串(`"25%"`);设 `0` 完全禁用预留;`>= 1` 恢复旧的完整预留行为(input + 用满预算的响应总能放进窗口 —— SGLang/vLLM 等严格后端的要求)。负数或无法解析的值会拒绝整个 `compress` 块。示例:窗口 262144 token、`max_tokens = 131072` → 默认 `0.25` 预留 65536 → 有效窗口 196608(旧完整预留:131072);`max_tokens = 65536` → 预留 65536 → 196608 不变(65536 ≤ 窗口的 25%)。与 billion-context-pi(`#207`)对齐,见 #896。

#### `maxContextLimit`

Expand Down
17 changes: 17 additions & 0 deletions src/config.ts
Original file line number Diff line number Diff line change
Expand Up @@ -324,6 +324,23 @@ export function resolveConfiguredContextLimit(
return undefined;
}

/** #924: the operator-declared max OUTPUT of a model (ModelEntry.output,
* documented as the model's max output size) — the mirror of
* resolveConfiguredContextLimit for the output-headroom fallback chain. When
* the request carries no output budget at all (Codex native Responses sends
* no max_output_tokens), this outranks the auto-fetched models.dev ceiling —
* same order as the window resolution (#344). */
export function resolveConfiguredOutputLimit(
routes: ProviderRoutes,
upstreamUrl: string | undefined,
model: string | undefined,
): number | undefined {
if (!model || !upstreamUrl) return undefined;
const m = findRoute(routes, upstreamUrl)?.models?.[model];
if (m?.output && m.output > 0) return m.output;
return undefined;
}

export function resolveCompressProtocol(routes: ProviderRoutes, upstreamUrl: string | undefined): "tools" | "marker" | undefined {
return findRoute(routes, upstreamUrl)?.compressProtocol;
}
Expand Down
44 changes: 39 additions & 5 deletions src/server.ts
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,8 @@ import { DEFAULT_STRIP_IMAGES_KEEP_RECENT, stripHistoricalImages } from "./strip
import type { ProxyOptions } from "./config.js";
import { loadOptions, loadRoutes } from "./config.js";
import { resetProxyCache } from "./upstream-proxy.js";
import { FALLBACK_EFFECTIVE_WINDOW_FLOOR, findRoute, lookupContextLimit, resolveConfiguredContextLimit, resolveCompressProtocol } from "./config.js";
import { contextFromRegistry, loadRegistry, peekRegistryContext } from "./registry.js";
import { FALLBACK_EFFECTIVE_WINDOW_FLOOR, findRoute, lookupContextLimit, resolveConfiguredContextLimit, resolveConfiguredOutputLimit, resolveCompressProtocol } from "./config.js";
import { contextFromRegistry, loadRegistry, peekRegistryContext, peekRegistryOutputLimit } from "./registry.js";
import { codexAlignedWindow } from "./codex-models.js";
import { fetchWithTimeout, MAX_REQUEST_BYTES, upstreamTimeoutMs } from "./fetch-util.js";
import { formatUpstreamError, getUpstreamConnectionStatus, recordUpstreamConnection, resolveProxy, resolveProxyDecision, proxyDispatcher, type UpstreamProxyDecision } from "./upstream-proxy.js";
Expand Down Expand Up @@ -586,6 +586,10 @@ function adminTrustedHosts(bindHost: string, port: number): Set<string> {
return set;
}

// #924: one-time-per-model log for the output-budget fallback (request carries
// no budget → configured/registry max output) — same pattern as windowSourceLogged.
const headroomFallbackLogged = new Set<string>();

async function handle(
req: http.IncomingMessage,
res: http.ServerResponse,
Expand Down Expand Up @@ -1422,8 +1426,10 @@ async function handle(
// exempt: its input limit is enforced independently of max_tokens
// (separate output budget), so reserving would shift every band down by
// maxOutput on every session for no safety gain — see
// shouldReserveOutputHeadroom. max_tokens is the exact output budget
// requested for THIS turn, so the reservation is precise and per-request.
// shouldReserveOutputHeadroom. The request's own budget field is the exact
// output budget requested for THIS turn, so it is precise and per-request;
// when the harness omits every budget field (#924 fallback below) the
// model's declared max output stands in for it.
// #896: headroomCap (compress.outputHeadroomMaxPct, default 0.25, aligned
// with billion-context-pi #207) caps the reservation at headroomCap × window
// — reserved = min(maxOutput, headroomCap × window). Replies longer than the
Expand All @@ -1436,7 +1442,35 @@ async function handle(
if (shouldReserveOutputHeadroom(protocol)) {
const p = parsed as Record<string, unknown>;
const rawMax = p.max_tokens ?? p.max_completion_tokens ?? p.max_output_tokens;
const maxOutput = typeof rawMax === "number" ? rawMax : 0;
let maxOutput = typeof rawMax === "number" ? rawMax : 0;
// #924: harnesses that omit every output-budget field (Codex native
// Responses sends no max_output_tokens — openai/codex#36180) still get
// the upstream's own default output cap applied, so reserving nothing
// leaves the nudge/truncate bands able to overflow with one long
// reply. Fall back to the model's declared max output — per-route
// config first (operator-declared, outranks auto-fetched data, same
// order as the window resolution #344), then the models.dev registry
// ceiling (cache-only + bundled-snapshot floor, never fetches — the
// source preflight's summary cap uses, #853) — through the SAME capped
// reservation below. Unknown model → 0 → today's behavior.
if (!(maxOutput > 0)) {
const fbModel = (parsed as { model?: string }).model;
if (fbModel) {
let host: string | undefined;
try { host = route?.rewrittenUrl ? new URL(route.rewrittenUrl).host : undefined; } catch { host = undefined; }
const cfgOut = resolveConfiguredOutputLimit(opts.routes, route?.rewrittenUrl, fbModel);
const regOut = peekRegistryOutputLimit(fbModel, host);
const budget = cfgOut !== undefined ? cfgOut : regOut;
if (typeof budget === "number" && budget > 0) {
maxOutput = budget;
const source = cfgOut !== undefined ? "configured" : "registry";
if (!headroomFallbackLogged.has(`${fbModel}|${source}`)) {
headroomFallbackLogged.add(`${fbModel}|${source}`);
log("info", `[headroom] model=${fbModel}: request carries no output budget; reserving against ${source} max output ${budget} (#924)`);
}
}
}
}
let reserved = reserveOutputHeadroom(reqConfig.modelContextLimit, maxOutput, headroomCap);
// Fallback-derived windows are optimistic guesses: never let the
// output-headroom reservation push the effective window below the
Expand Down
Loading
Loading