From eb8bd3ccb1dabaf77be1f39b80f2523e012c9c12 Mon Sep 17 00:00:00 2001 From: ranxianglei Date: Sat, 7 Nov 2026 21:18:51 +0800 Subject: [PATCH] =?UTF-8?q?fix(#5199):=20network.throttle429Scope=20?= =?UTF-8?q?=E2=80=94=20model-scoped=20holds=20for=20marker-less=20gateway?= =?UTF-8?q?=20429s?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Implements #5853 decision 1 option A (recommended); decision 2 unchanged (streak-1 arming kept). Default "credential" preserves #5696 semantics — zero behavior change without opt-in. - knobs: BILI_THROTTLE_429_SCOPE > network.throttle429Scope > "credential", read per settlement (hot flip) - upstream-throttle: throttle429ModelScope() — recognized KeyRing/bili-relay markers win as before; scope=model settles marker-less 429s (RPM/quota text, opaque relay rewrites) on the request model; model-less sends stay credential-wide - relay/loop/degenerate-retry lanes settle through the wrapper - docs: CONFIGURATION en/zh + website seed regenerated, LIFECYCLE env row - tests: tests/issue5199-throttle429-scope.test.ts (6 rows, both directions) --- CONFIGURATION.md | 2 + CONFIGURATION.zh-CN.md | 2 + LIFECYCLE.md | 2 +- src/config.ts | 7 ++ src/degenerate-retry.ts | 4 +- src/knobs.ts | 17 +++ src/loop/core.ts | 6 +- src/server/relay.ts | 4 +- src/upstream-throttle.ts | 22 ++++ tests/issue5199-throttle429-scope.test.ts | 140 ++++++++++++++++++++++ website/config-reference/process.yaml | 11 +- website/docs/index.html | 2 +- website/main.js | 4 +- 13 files changed, 210 insertions(+), 13 deletions(-) create mode 100644 tests/issue5199-throttle429-scope.test.ts diff --git a/CONFIGURATION.md b/CONFIGURATION.md index 2472573fd7..2a67f59730 100644 --- a/CONFIGURATION.md +++ b/CONFIGURATION.md @@ -171,6 +171,7 @@ This index is generated from `website/config-reference/*.yaml` — edit the seed | `network.throttle429MaxConsecutive` | number | 5 | BILI_THROTTLE_429_MAX_CONSECUTIVE | Consecutive upstream 429s before the hold turns terminal (local 424 outside every known SDK retry set); one probe per backoff window keeps checking until any non-429 response clears an account hold, or the matching model's non-429 response clears its model hold. | | `network.throttle429BaseDelayMs` | number | 2000 | BILI_THROTTLE_429_BASE_DELAY_MS | Base backoff for the 429 hold (doubles per consecutive 429, jittered into [0.5×, 1×], capped by throttle429MaxDelayMs; an upstream Retry-After wins when larger). | | `network.throttle429MaxDelayMs` | number | 30000 | BILI_THROTTLE_429_MAX_DELAY_MS | Ceiling for the 429 hold backoff and for the terminal-phase probe interval. | +| `network.throttle429Scope` | string | credential | BILI_THROTTLE_429_SCOPE | Hold granularity for marker-less upstream 429s (#5199). `credential` (default) keeps #5696's conservative semantics — any 429 without a recognized model marker (`keyring_model_temporarily_unavailable`, or a bili relay's `bili_throttle_hold` scope=model) holds the whole origin+credential, so every model behind that credential waits out the backoff. `model` keys the hold by the send's request model instead, for gateways that front many independently-limited models behind one credential (LiteLLM/one-api/opencode.ai-style relays where one model's RPM exhaustion must not freeze its siblings); requests whose model cannot be identified remain credential-wide, and recognized markers keep their own semantics in both modes. | | `network.maxShrinkPerCompress` | number | unset | BILI_MAX_SHRINK_PER_COMPRESS | Cap on how much one compression may shrink history (anti-collapse guard). | | `network.proxyKeepAliveMaxMs` | number | 55000 (0 = one-shot connections) | BILI_PROXY_KEEPALIVE_MAX_MS | Max lifetime of a connection through an outbound egress proxy before forcing reconnection (avoids stale NAT mappings). | | `network.postResponseLingerMs` | number | 5000 | BILI_POST_RESPONSE_LINGER_MS | Graceful-close budget for proxy-initiated closes after the response completed: the socket is held waiting for the peer's FIN/TLS close_notify and destroyed at the deadline (reason=linger-backstop). | @@ -1980,6 +1981,7 @@ File keys resolve only when the matching env var is unset. Defaults in parenthes | `BILI_THROTTLE_429_BASE_DELAY_MS` | `network.throttle429BaseDelayMs` | 2000 | | `BILI_THROTTLE_429_MAX_CONSECUTIVE` | `network.throttle429MaxConsecutive` | 5 | | `BILI_THROTTLE_429_MAX_DELAY_MS` | `network.throttle429MaxDelayMs` | 30000 | +| `BILI_THROTTLE_429_SCOPE` | `network.throttle429Scope` | credential | | `BILI_UPDATE_CHECK_INTERVAL_MS` | `update.checkIntervalMs` | 180000 | | `BILI_UPDATE_POLICY_URL` | `update.policyUrl` | official repo raw URL | | `BILI_UPDATE_REGISTRY` | `update.registry` | "npmjs" (registry.npmjs.org) | diff --git a/CONFIGURATION.zh-CN.md b/CONFIGURATION.zh-CN.md index d5c911254c..fb88c608e9 100644 --- a/CONFIGURATION.zh-CN.md +++ b/CONFIGURATION.zh-CN.md @@ -171,6 +171,7 @@ | `network.throttle429MaxConsecutive` | number | 5 | BILI_THROTTLE_429_MAX_CONSECUTIVE | 连续上游 429 达到该次数后 hold 转为终局(本地 424,不在任何已知 SDK 重试集合内);每个退避窗口仍放行一次探测,直到账号级 hold 收到任意非 429 响应,或模型级 hold 收到同一模型的非 429 响应后清除。 | | `network.throttle429BaseDelayMs` | number | 2000 | BILI_THROTTLE_429_BASE_DELAY_MS | 429 hold 的基础退避(每连续一次 429 翻倍、抖动到 [0.5×, 1×]、以 throttle429MaxDelayMs 封顶;上游 Retry-After 更大时优先)。 | | `network.throttle429MaxDelayMs` | number | 30000 | BILI_THROTTLE_429_MAX_DELAY_MS | 429 hold 退避的上限,也是终局阶段探测间隔的上限。 | +| `network.throttle429Scope` | string | credential | BILI_THROTTLE_429_SCOPE | 无标记上游 429 的 hold 粒度(#5199)。`credential`(默认)保持 #5696 的保守语义——凡未携带可识别模型标记(`keyring_model_temporarily_unavailable`,或 bili 中继的 `bili_throttle_hold` scope=model)的 429 均按整个 origin+凭据持有,该凭据下所有模型一起退避等待。`model` 改按发送请求的模型分桶持有,适用于单凭据背后多模型独立限流的网关(LiteLLM/one-api/opencode.ai 类中继,一个模型的 RPM 耗尽不应冻结兄弟模型);无法识别模型的请求仍按凭据级持有,两种模式下已识别标记的语义均不变。 | | `network.maxShrinkPerCompress` | number | unset | BILI_MAX_SHRINK_PER_COMPRESS | 单次压缩允许收缩历史的上限(防塌缩护栏)。 | | `network.proxyKeepAliveMaxMs` | number | 55000 (0 = one-shot connections) | BILI_PROXY_KEEPALIVE_MAX_MS | 经出站代理的连接最大存活时长,到期强制重建(规避陈旧 NAT 映射)。 | | `network.postResponseLingerMs` | number | 5000 | BILI_POST_RESPONSE_LINGER_MS | 代理主动发起的响应后关闭的优雅关闭预算:socket 挂起等待对端 FIN/TLS close_notify,到点销毁(reason=linger-backstop)。 | @@ -1957,6 +1958,7 @@ ACP 原生 agent(当前为 `pi` 扩展)会在每个进程内向代理上报 | `BILI_THROTTLE_429_BASE_DELAY_MS` | `network.throttle429BaseDelayMs` | 2000 | | `BILI_THROTTLE_429_MAX_CONSECUTIVE` | `network.throttle429MaxConsecutive` | 5 | | `BILI_THROTTLE_429_MAX_DELAY_MS` | `network.throttle429MaxDelayMs` | 30000 | +| `BILI_THROTTLE_429_SCOPE` | `network.throttle429Scope` | credential | | `BILI_UPDATE_CHECK_INTERVAL_MS` | `update.checkIntervalMs` | 180000 | | `BILI_UPDATE_POLICY_URL` | `update.policyUrl` | official repo raw URL | | `BILI_UPDATE_REGISTRY` | `update.registry` | "npmjs" (registry.npmjs.org) | diff --git a/LIFECYCLE.md b/LIFECYCLE.md index 5ecdd0d0ca..fd9bf1a640 100644 --- a/LIFECYCLE.md +++ b/LIFECYCLE.md @@ -96,7 +96,7 @@ | `BILI_EXPOSURE_LOG_INTERVAL_MS` | 3600000(0 禁用,垃圾值回落 3600000) | | `BILI_MAX_SHRINK_PER_COMPRESS` | unset(不做收缩引导) | | `BILI_REPLAY_RETRY_MAX` / `BILI_REPLAY_RETRY_BASE_MS` | 3 / 1500 | -| `BILI_THROTTLE_429` / `BILI_THROTTLE_429_MAX_CONSECUTIVE` / `BILI_THROTTLE_429_BASE_DELAY_MS` / `BILI_THROTTLE_429_MAX_DELAY_MS` | true / 5 / 2000 / 30000(#4204 出站 429 治理器:同一 origin+凭据连续上游 429 后本地 hold、零出站,退避×抖动、尊重 Retry-After;达上限转终局 424(不在各 SDK 重试集合内),每窗口一次探测自动恢复,任何非 429 响应即清除;`false` 恢复旧的总是转发行为) | +| `BILI_THROTTLE_429` / `BILI_THROTTLE_429_MAX_CONSECUTIVE` / `BILI_THROTTLE_429_BASE_DELAY_MS` / `BILI_THROTTLE_429_MAX_DELAY_MS` / `BILI_THROTTLE_429_SCOPE` | true / 5 / 2000 / 30000 / credential(#4204 出站 429 治理器:同一 origin+凭据连续上游 429 后本地 hold、零出站,退避×抖动、尊重 Retry-After;达上限转终局 424(不在各 SDK 重试集合内),每窗口一次探测自动恢复,任何非 429 响应即清除;`false` 恢复旧的总是转发行为。#5199:scope=`model` 时无标记 429 按请求模型分桶 hold——单凭据多模型独立限流网关(LiteLLM/one-api 类)一个模型的 RPM 耗尽不再冻结兄弟模型;无法识别模型的请求仍按凭据级) | | `BILI_STREAM_KEEPALIVE_MS` | 15000(0 禁用 hold) | | `BILI_PREFLIGHT_HOLD_MS` | 30000 | | `BILI_PREFLIGHT_DEAD_END_COOLDOWN_MS` | 300000 | diff --git a/src/config.ts b/src/config.ts index a3591e00d2..a519e34e55 100644 --- a/src/config.ts +++ b/src/config.ts @@ -2264,6 +2264,13 @@ type FileConfig = { /** 429-hold backoff ceiling in ms; also the probe cadence once the hold is terminal (#4204; * was env-only BILI_THROTTLE_429_MAX_DELAY_MS; default 30000). */ throttle429MaxDelayMs?: number; + /** Hold granularity for marker-less upstream 429s (#5199; env BILI_THROTTLE_429_SCOPE; default "credential"). + * "credential" — #5696's conservative default: any 429 without a recognized model marker holds the whole + * origin+credential (every model, every session on it). "model" — hold only the send's request model, for + * gateways fronting many independently-limited models behind one credential (LiteLLM/one-api/opencode.ai-style + * relays); requests without an identifiable model remain credential-wide, and recognized KeyRing markers keep + * their own semantics in both modes. */ + throttle429Scope?: "credential" | "model"; /** Per-compress shrink steering factor in (0,1] (was BILI_MAX_SHRINK_PER_COMPRESS; default unset). */ maxShrinkPerCompress?: number; /** Outbound proxy keep-alive ceiling, 0 for one-shot (was BILI_PROXY_KEEPALIVE_MAX_MS; default 55000). */ diff --git a/src/degenerate-retry.ts b/src/degenerate-retry.ts index ab09d676ad..a0e04398a8 100644 --- a/src/degenerate-retry.ts +++ b/src/degenerate-retry.ts @@ -1,5 +1,5 @@ import { fetchWithTimeout } from "./fetch-util.js"; -import { beginProvisionalUpstreamThrottle429, keyringModel429Scope, noteUpstreamThrottleResolved, releaseUpstreamThrottleProbe, requestModelFromBody, reserveUpstreamThrottleClassification, upstreamThrottleGate, type ThrottleClassificationReservation } from "./upstream-throttle.js"; +import { beginProvisionalUpstreamThrottle429, noteUpstreamThrottleResolved, releaseUpstreamThrottleProbe, requestModelFromBody, reserveUpstreamThrottleClassification, throttle429ModelScope, upstreamThrottleGate, type ThrottleClassificationReservation } from "./upstream-throttle.js"; import { hasCompactionTrigger } from "./codex-compact.js"; import { proxyDispatcher } from "./upstream-proxy.js"; import type { WireProtocol } from "./util.js"; @@ -191,7 +191,7 @@ export function makeContinuationRefetch(opts: ContinuationRetryOpts): () => Prom // is the continuation callback's 250ms latency budget. const bodyRead = readSmallThrottleErrorBody(r.response) .then((errorBody) => { - const modelScope = keyringModel429Scope(r.response.status, requestModel, errorBody); + const modelScope = throttle429ModelScope(r.response.status, requestModel, errorBody); settle(modelScope); }) .catch(() => { diff --git a/src/knobs.ts b/src/knobs.ts index e86b6ac92e..e9a0f3b954 100644 --- a/src/knobs.ts +++ b/src/knobs.ts @@ -145,6 +145,23 @@ export function throttle429MaxDelayMs(): number { return Number.isFinite(v) && v >= 0 ? v : THROTTLE_429_MAX_DELAY_MS; } +/** BILI_THROTTLE_429_SCOPE > network.throttle429Scope > "credential". Hold + * granularity for marker-less upstream 429s (#5199/#5853 decision 1, option + * A): "credential" keeps #5696's conservative default — an unrecognized 429 + * body (RPM/quota text, opaque relay rewrites) holds the whole + * origin+credential, freezing every model behind it; "model" keys that hold + * by the send's request model, for gateways that front many independently + * limited models behind one credential (LiteLLM/one-api/opencode.ai-style + * relays). Recognized KeyRing/bili-relay markers keep their own semantics in + * both modes; requests whose model cannot be identified stay + * credential-wide, so worst-case protection never widens beyond legacy. + * Read per settlement so the switch flips live. */ +export function throttle429Scope(): "credential" | "model" { + const env = process.env.BILI_THROTTLE_429_SCOPE; + const raw = env !== undefined && env !== "" ? env : fileNetwork().throttle429Scope; + return raw === "model" ? "model" : "credential"; +} + /** BILI_MAX_SHRINK_PER_COMPRESS > network.maxShrinkPerCompress > unset (no steering). */ export function maxShrinkPerCompress(): number | undefined { const v = tNum("BILI_MAX_SHRINK_PER_COMPRESS", fileNetwork().maxShrinkPerCompress); diff --git a/src/loop/core.ts b/src/loop/core.ts index fec1671a7b..d107c9ae04 100644 --- a/src/loop/core.ts +++ b/src/loop/core.ts @@ -33,7 +33,7 @@ import { applyConfiguredCompression } from "../external-summary-compress.js"; import { executeSearchContextTarget, resolveDecompress } from "../decompress-shared.js"; import { toolFail, type ProxyToolResult } from "../proxy-tool-result.js"; import { fetchWithRetry, UpstreamHttpError } from "../fetch-util.js"; -import { keyringModel429Scope, noteUpstreamThrottle429, noteUpstreamThrottleResolved } from "../upstream-throttle.js"; +import { noteUpstreamThrottle429, noteUpstreamThrottleResolved, throttle429ModelScope } from "../upstream-throttle.js"; import { classifyUpstreamFailure, type SkippedProxyInfo, type UpstreamFailureKind } from "../upstream-fail.js"; import { formatUpstreamError, proxyDispatcher } from "../upstream-proxy.js"; import { warnCacheCollapse } from "../cache-warn.js"; @@ -601,7 +601,7 @@ export async function* runCompressLoop( signal, (info) => { if (info.status === 429) { - const modelScope = keyringModel429Scope(info.status, requestModel, info.detail); + const modelScope = throttle429ModelScope(info.status, requestModel, info.detail); noteUpstreamThrottle429(requestOptions.url, requestOptions.headers, null, Date.now(), modelScope); // Retry-After is not carried on replay metadata. } // #189: correlate the rejection with the rewrite that @@ -622,7 +622,7 @@ export async function* runCompressLoop( return upstreamSend; } catch (error) { if (error instanceof UpstreamHttpError && error.status === 429) { - const modelScope = keyringModel429Scope(error.status, requestModel, error.body); + const modelScope = throttle429ModelScope(error.status, requestModel, error.body); noteUpstreamThrottle429(requestOptions.url, requestOptions.headers, null, Date.now(), modelScope); } throw error; diff --git a/src/server/relay.ts b/src/server/relay.ts index 2783b4741a..112e614ee5 100644 --- a/src/server/relay.ts +++ b/src/server/relay.ts @@ -58,7 +58,7 @@ import { safePrefix, safeSuffix, scrubLoneSurrogatesOnWire } from "../text-safe. import { clearUpstreamAlertsForHost, noteCircuitFailFastLog, recordUpstreamAlert, recordUpstreamCircuitFailure, recordUpstreamCircuitSuccess, upstreamCircuitBlockReason, upstreamCircuitLastFailureKind, upstreamCircuitRetryAfterSeconds } from "../upstream-alerts.js"; import { describeUpstreamFailure, formatUpstreamError, proxyDispatcher, recordUpstreamConnection } from "../upstream-proxy.js"; import { markExpectedFailFast, withUpstreamFailure, type UpstreamFailureKind } from "../upstream-fail.js"; -import { keyringModel429Scope, noteUpstreamThrottle429, noteUpstreamThrottleResolved, releaseUpstreamThrottleProbe, requestModelFromBody, type ThrottleLocalReject, upstreamThrottleGate, upstreamThrottlePeekReject } from "../upstream-throttle.js"; +import { noteUpstreamThrottle429, noteUpstreamThrottleResolved, releaseUpstreamThrottleProbe, requestModelFromBody, throttle429ModelScope, type ThrottleLocalReject, upstreamThrottleGate, upstreamThrottlePeekReject } from "../upstream-throttle.js"; import { applyEstimateCalibration, currentCalibrationFactor, inspectContextOverflow, isMarkerlessOverflowReject, MIN_ASSUMED_WINDOW_TOKENS, normalizeUpstreamOrigin, usageOutputTotal, usageTotals, type ContextOverflowInfo, type WireProtocol } from "../util.js"; import { billOutputUsage } from "../billing.js"; import { DUMP_PLAINTEXT_BANNER } from "../dump-banner.js"; @@ -1691,7 +1691,7 @@ async function forwardCaptured( // keyring_all_limited and every ambiguous response remain account-wide. if (req.method !== "GET" && req.method !== "HEAD") { if (upstream.status === 429) { - const modelScope = keyringModel429Scope(upstream.status, throttleModel, errBody); + const modelScope = throttle429ModelScope(upstream.status, throttleModel, errBody); noteUpstreamThrottle429(upstreamUrl, headers, upstream.headers.get("retry-after"), Date.now(), modelScope, throttleProbeToken); } else { noteUpstreamThrottleResolved(upstreamUrl, headers, upstream.status, Date.now(), throttleModel, throttleProbeToken); diff --git a/src/upstream-throttle.ts b/src/upstream-throttle.ts index 197e3d2a61..9efb5e85c7 100644 --- a/src/upstream-throttle.ts +++ b/src/upstream-throttle.ts @@ -35,6 +35,7 @@ import { throttle429Enabled, throttle429MaxConsecutive, throttle429MaxDelayMs, + throttle429Scope, } from "./knobs.js"; /** Bounded state table (same posture as MAX_CIRCUITS in upstream-alerts.ts). */ @@ -241,6 +242,27 @@ export function keyringModel429Scope( return undefined; } +/** + * The hold scope for an observed 429, combining #5696's marker discipline with + * the #5199/#5853 knob: a recognized KeyRing / bili-relay marker wins exactly + * as before; otherwise, when network.throttle429Scope is "model" and the send + * carried an identifiable request model, a marker-less gateway 429 (RPM/quota + * text, opaque relay-rewritten bodies — LiteLLM/one-api/opencode.ai shapes + * where one model's pool exhaustion must not freeze its siblings) settles + * model-scoped instead of credential-wide. Model-less sends keep the legacy + * credential key, so worst-case protection never widens beyond legacy. + */ +export function throttle429ModelScope( + status: number, + requestModel: string | undefined, + responseBody: string | Uint8Array | null | undefined, +): string | undefined { + const recognized = keyringModel429Scope(status, requestModel, responseBody); + if (recognized !== undefined) return recognized; + if (status !== 429 || throttle429Scope() !== "model") return undefined; + return requestModel !== undefined && requestModel.length > 0 ? requestModel : undefined; +} + function modelHoldKey(key: string, model: string): string { return `${key}\u0000model\u0000${model}`; } diff --git a/tests/issue5199-throttle429-scope.test.ts b/tests/issue5199-throttle429-scope.test.ts new file mode 100644 index 0000000000..5466001395 --- /dev/null +++ b/tests/issue5199-throttle429-scope.test.ts @@ -0,0 +1,140 @@ +// #5199: marker-less gateway 429s are credential-wide by default (#5696 +// semantics); network.throttle429Scope="model" keys them by the request model +// instead, so one model's RPM/quota exhaustion on a multi-pool gateway +// (LiteLLM/one-api/opencode.ai-style relays) cannot freeze its siblings. +// Recognized KeyRing/bili-relay markers keep their own semantics in both modes. +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { throttle429Scope } from "../src/knobs.ts"; +import { + activeThrottleHolds, + noteUpstreamThrottle429, + noteUpstreamThrottleResolved, + throttle429ModelScope, + upstreamThrottleGate, + _resetUpstreamThrottleForTest, +} from "../src/upstream-throttle.ts"; + +const URL_A = "http://gw.example/v1/chat/completions"; +const H_CRED_A: Record = { authorization: "Bearer AAA" }; +const T0 = 1_000_000; +const MODEL = "deepseek-v4.1-flash"; +const SIBLING = "glm-5.6-flash"; +// The two field-report shapes: an aggregator gateway RPM body with no +// structured marker (#5199, monkey-sking) and an opaque relay-rewritten +// prefill-cap refusal (opencode.ai lane, magpie log). +const MARKERLESS_RPM = JSON.stringify({ error: { code: null, message: "qnaigc: rate limit reached for RPM (request_id: req-1)" } }); +const MARKERLESS_PREFILL = JSON.stringify({ error: { message: "[prefill_cap_refused] the cell is at capacity; retry shortly (after 30 s)" } }); +const KEYRING_MODEL_ERROR = JSON.stringify({ error: { code: "keyring_model_temporarily_unavailable", model: MODEL } }); + +function throttleReject(url: string, headers: Record, now?: number, requestModel?: string) { + return upstreamThrottleGate(url, headers, now, requestModel).reject; +} + +const KNOB_ENVS = [ + "BILI_THROTTLE_429", + "BILI_THROTTLE_429_MAX_CONSECUTIVE", + "BILI_THROTTLE_429_BASE_DELAY_MS", + "BILI_THROTTLE_429_MAX_DELAY_MS", + "BILI_THROTTLE_429_SCOPE", +] as const; + +// Env beats the config file — pin every knob explicitly so the daemon's own +// ~/.config/billion-context/billion-context.json can never leak into a row. +function withKnobs(vars: Partial>, fn: () => void): void { + const saved: Record = {}; + for (const k of KNOB_ENVS) saved[k] = process.env[k]; + try { + for (const [k, v] of Object.entries(vars)) process.env[k] = v; + fn(); + } finally { + for (const k of KNOB_ENVS) { + if (saved[k] === undefined) delete process.env[k]; + else process.env[k] = saved[k]; + } + } +} + +test("#5199: knob reads env live and rejects unknown values back to credential", () => { + withKnobs({}, () => { + assert.equal(throttle429Scope(), "credential", "default is credential"); + process.env.BILI_THROTTLE_429_SCOPE = "model"; + assert.equal(throttle429Scope(), "model", "env flips per read, no restart"); + process.env.BILI_THROTTLE_429_SCOPE = "Model"; + assert.equal(throttle429Scope(), "credential", "value match is exact"); + process.env.BILI_THROTTLE_429_SCOPE = ""; + assert.equal(throttle429Scope(), "credential", "empty env reads as unset"); + }); +}); + +test("#5199: default scope keeps marker-less gateway 429s credential-wide (#5696 row)", () => { + withKnobs({ BILI_THROTTLE_429: "1" }, () => { + assert.equal(throttle429ModelScope(429, MODEL, MARKERLESS_RPM), undefined); + assert.equal(throttle429ModelScope(429, MODEL, MARKERLESS_PREFILL), undefined); + assert.equal(throttle429ModelScope(429, MODEL, undefined), undefined); + assert.equal(throttle429ModelScope(429, MODEL, "{malformed"), undefined); + // The recognized marker keeps granting model scope in the default mode. + assert.equal(throttle429ModelScope(429, MODEL, KEYRING_MODEL_ERROR), MODEL); + }); +}); + +test("#5199: scope=model settles marker-less 429s on the request model", () => { + withKnobs({ BILI_THROTTLE_429: "1", BILI_THROTTLE_429_SCOPE: "model" }, () => { + assert.equal(throttle429ModelScope(429, MODEL, MARKERLESS_RPM), MODEL); + assert.equal(throttle429ModelScope(429, MODEL, MARKERLESS_PREFILL), MODEL); + assert.equal(throttle429ModelScope(429, MODEL, "{malformed"), MODEL, "malformed body cannot identify a wider scope than the send itself"); + // Negative rows: no identifiable request model stays credential-wide. + assert.equal(throttle429ModelScope(429, undefined, MARKERLESS_RPM), undefined); + assert.equal(throttle429ModelScope(429, "", MARKERLESS_RPM), undefined); + // Non-429 never carries a scope. + assert.equal(throttle429ModelScope(400, MODEL, MARKERLESS_RPM), undefined); + // The recognized marker still wins (identical outcome, marker discipline). + assert.equal(throttle429ModelScope(429, MODEL, KEYRING_MODEL_ERROR), MODEL); + }); +}); + +test("#5199: scope=model — a marker-less 429 holds its own model, siblings keep sending", () => { + withKnobs({ BILI_THROTTLE_429: "1", BILI_THROTTLE_429_SCOPE: "model" }, () => { + _resetUpstreamThrottleForTest(); + noteUpstreamThrottle429(URL_A, H_CRED_A, null, T0, throttle429ModelScope(429, MODEL, MARKERLESS_RPM)); + assert.equal(activeThrottleHolds()[0].model, MODEL, "the hold is model-scoped"); + assert.ok(throttleReject(URL_A, H_CRED_A, T0 + 100, MODEL), "the exhausted model is held"); + assert.equal(throttleReject(URL_A, H_CRED_A, T0 + 100, SIBLING), undefined, "a sibling model on the same credential stays eligible (#5199 fix)"); + assert.equal(throttleReject(URL_A, H_CRED_A, T0 + 100, undefined), undefined, "model-less sends bypass a model hold (a model mismatch cannot freeze them)"); + + noteUpstreamThrottleResolved(URL_A, H_CRED_A, 200, T0 + 200, SIBLING); + assert.ok(throttleReject(URL_A, H_CRED_A, T0 + 300, MODEL), "a sibling's success does not clear the model hold"); + noteUpstreamThrottleResolved(URL_A, H_CRED_A, 200, T0 + 400, MODEL); + assert.equal(throttleReject(URL_A, H_CRED_A, T0 + 500, MODEL), undefined, "the matching model's success clears it"); + _resetUpstreamThrottleForTest(); + }); +}); + +test("#5199: scope=model — a marker-less model-LESS 429 stays credential-wide (protection never widens)", () => { + withKnobs({ BILI_THROTTLE_429: "1", BILI_THROTTLE_429_SCOPE: "model" }, () => { + _resetUpstreamThrottleForTest(); + noteUpstreamThrottle429(URL_A, H_CRED_A, null, T0, throttle429ModelScope(429, undefined, MARKERLESS_RPM)); + const st = activeThrottleHolds()[0]; + assert.equal(st.model, undefined, "no request model → legacy credential-wide hold"); + assert.ok(throttleReject(URL_A, H_CRED_A, T0 + 100, MODEL), "credential hold still gates a model send"); + assert.ok(throttleReject(URL_A, H_CRED_A, T0 + 100, SIBLING), "credential hold still gates every model (#4204 storm protection intact)"); + _resetUpstreamThrottleForTest(); + }); +}); + +test("#5199: scope flip is read per settlement — credential-mode account hold stays until cleared", () => { + withKnobs({ BILI_THROTTLE_429: "1" }, () => { + _resetUpstreamThrottleForTest(); + // Armed while scope=credential: a credential-wide hold. + noteUpstreamThrottle429(URL_A, H_CRED_A, null, T0, throttle429ModelScope(429, MODEL, MARKERLESS_RPM)); + assert.ok(throttleReject(URL_A, H_CRED_A, T0 + 100, SIBLING)); + // Flipping the knob live changes only FUTURE settlements... + process.env.BILI_THROTTLE_429_SCOPE = "model"; + assert.equal(throttle429ModelScope(429, SIBLING, MARKERLESS_RPM), SIBLING); + // ...the already-armed credential hold keeps gating until its probe clears. + assert.ok(throttleReject(URL_A, H_CRED_A, T0 + 150, SIBLING)); + noteUpstreamThrottleResolved(URL_A, H_CRED_A, 200, T0 + 200, SIBLING); + assert.equal(throttleReject(URL_A, H_CRED_A, T0 + 250, SIBLING), undefined, "cleared as before"); + _resetUpstreamThrottleForTest(); + }); +}); diff --git a/website/config-reference/process.yaml b/website/config-reference/process.yaml index c67838b313..2993db1e22 100644 --- a/website/config-reference/process.yaml +++ b/website/config-reference/process.yaml @@ -10,8 +10,8 @@ entries: site_group: proc desc_en: "Transport timeouts, watchdogs and replay policy for upstream calls." desc_zh: "上游调用的传输超时、看门狗与重放策略。" - site_en: "network — transport timeouts, replays & the 429 egress governor: upstreamTimeoutMs 720000 · requestWatchdogMs 2× · replayRetryMax 3 · replayRetryBaseMs 1500 · keepAliveTimeoutMs 5000 · clientErrorBackstopMs 30000 · exposureLogIntervalMs 3600000 · streamKeepAliveMs 15000 · preflightHoldMs 30000 · preflightDeadEndCooldownMs 300000 · proxyKeepAliveMaxMs 55000 · throttle429Enabled true · throttle429MaxConsecutive 5 · throttle429BaseDelayMs 2000 · throttle429MaxDelayMs 30000." - site_zh: "network —— 传输超时、重放与 429 出站治理:upstreamTimeoutMs 720000 · requestWatchdogMs 2× · replayRetryMax 3 · replayRetryBaseMs 1500 · keepAliveTimeoutMs 5000 · clientErrorBackstopMs 30000 · exposureLogIntervalMs 3600000 · streamKeepAliveMs 15000 · preflightHoldMs 30000 · preflightDeadEndCooldownMs 300000 · proxyKeepAliveMaxMs 55000 · throttle429Enabled true · throttle429MaxConsecutive 5 · throttle429BaseDelayMs 2000 · throttle429MaxDelayMs 30000。" + site_en: "network — transport timeouts, replays & the 429 egress governor: upstreamTimeoutMs 720000 · requestWatchdogMs 2× · replayRetryMax 3 · replayRetryBaseMs 1500 · keepAliveTimeoutMs 5000 · clientErrorBackstopMs 30000 · exposureLogIntervalMs 3600000 · streamKeepaliveMs 15000 · preflightHoldMs 30000 · preflightDeadEndCooldownMs 300000 · proxyKeepAliveMaxMs 55000 · throttle429Enabled true · throttle429MaxConsecutive 5 · throttle429BaseDelayMs 2000 · throttle429MaxDelayMs 30000 · throttle429Scope credential." + site_zh: "network —— 传输超时、重放与 429 出站治理:upstreamTimeoutMs 720000 · requestWatchdogMs 2× · replayRetryMax 3 · replayRetryBaseMs 1500 · keepAliveTimeoutMs 5000 · clientErrorBackstopMs 30000 · exposureLogIntervalMs 3600000 · streamKeepAliveMs 15000 · preflightHoldMs 30000 · preflightDeadEndCooldownMs 300000 · proxyKeepAliveMaxMs 55000 · throttle429Enabled true · throttle429MaxConsecutive 5 · throttle429BaseDelayMs 2000 · throttle429MaxDelayMs 30000 · throttle429Scope credential。" - key: network.upstreamTimeoutMs type: number default: "720000" @@ -110,6 +110,13 @@ entries: group: process desc_en: "Ceiling for the 429 hold backoff and for the terminal-phase probe interval." desc_zh: "429 hold 退避的上限,也是终局阶段探测间隔的上限。" + - key: network.throttle429Scope + type: string + default: "credential" + env: BILI_THROTTLE_429_SCOPE + group: process + desc_en: "Hold granularity for marker-less upstream 429s (#5199). `credential` (default) keeps #5696's conservative semantics — any 429 without a recognized model marker (`keyring_model_temporarily_unavailable`, or a bili relay's `bili_throttle_hold` scope=model) holds the whole origin+credential, so every model behind that credential waits out the backoff. `model` keys the hold by the send's request model instead, for gateways that front many independently-limited models behind one credential (LiteLLM/one-api/opencode.ai-style relays where one model's RPM exhaustion must not freeze its siblings); requests whose model cannot be identified remain credential-wide, and recognized markers keep their own semantics in both modes." + desc_zh: "无标记上游 429 的 hold 粒度(#5199)。`credential`(默认)保持 #5696 的保守语义——凡未携带可识别模型标记(`keyring_model_temporarily_unavailable`,或 bili 中继的 `bili_throttle_hold` scope=model)的 429 均按整个 origin+凭据持有,该凭据下所有模型一起退避等待。`model` 改按发送请求的模型分桶持有,适用于单凭据背后多模型独立限流的网关(LiteLLM/one-api/opencode.ai 类中继,一个模型的 RPM 耗尽不应冻结兄弟模型);无法识别模型的请求仍按凭据级持有,两种模式下已识别标记的语义均不变。" - key: network.maxShrinkPerCompress type: number default: "unset" diff --git a/website/docs/index.html b/website/docs/index.html index e6a2ca67ac..7e306202b0 100644 --- a/website/docs/index.html +++ b/website/docs/index.html @@ -293,7 +293,7 @@

Other top-level fields

  • mitm.enabled / mitm.domains — HTTPS interception switch and whitelist domains for login clients.
  • mitm.handshakeTimeoutMs — TLS handshake timeout (default 10000).
  • Process blocks
  • -
  • network — transport timeouts, replays & the 429 egress governor: upstreamTimeoutMs 720000 · requestWatchdogMs 2× · replayRetryMax 3 · replayRetryBaseMs 1500 · keepAliveTimeoutMs 5000 · clientErrorBackstopMs 30000 · exposureLogIntervalMs 3600000 · streamKeepAliveMs 15000 · preflightHoldMs 30000 · preflightDeadEndCooldownMs 300000 · proxyKeepAliveMaxMs 55000 · throttle429Enabled true · throttle429MaxConsecutive 5 · throttle429BaseDelayMs 2000 · throttle429MaxDelayMs 30000.
  • +
  • network — transport timeouts, replays & the 429 egress governor: upstreamTimeoutMs 720000 · requestWatchdogMs 2× · replayRetryMax 3 · replayRetryBaseMs 1500 · keepAliveTimeoutMs 5000 · clientErrorBackstopMs 30000 · exposureLogIntervalMs 3600000 · streamKeepaliveMs 15000 · preflightHoldMs 30000 · preflightDeadEndCooldownMs 300000 · proxyKeepAliveMaxMs 55000 · throttle429Enabled true · throttle429MaxConsecutive 5 · throttle429BaseDelayMs 2000 · throttle429MaxDelayMs 30000 · throttle429Scope credential.
  • guardrails — pi/omp plugin tool-output + stream text-run guardrails: bashTimeout 60 · maxOutputBytes 50000 · repetition true · textRunThreshold 200.
  • persist — session persistence to disk: enabled true · zstd false · debounceMs 500 · tailTokens 16384 · epermAlertThreshold 5 · epermAlertRepeatMs 0.
  • storage — unified storage sidecar: unified legacy — file tier selects the lane-wide storage mode; env BILI_STORAGE_UNIFIED wins when set.
  • diff --git a/website/main.js b/website/main.js index 6fcab674d2..eaca3f6e12 100644 --- a/website/main.js +++ b/website/main.js @@ -132,7 +132,7 @@ "tt.ccx": "codexCompact — intercept | pass (default intercept): handle Codex's compaction requests in bili or forward upstream — explicit forms plus the inline auto-compact transiting as an ordinary turn (prompt-form detection, #3008; overridden compact_prompt transits verbatim).", "ttl.ttl": "ccrRetrievalTtlMs — lifetime of CCR retrieval pointers (default 600000; 0 disables retrieval).", "ttl.tmp": "decompressTmpCap — size cap for temporary decompressed blocks (default 50).", - "tt.network": "network — transport timeouts, replays & the 429 egress governor: upstreamTimeoutMs 720000 · requestWatchdogMs 2× · replayRetryMax 3 · replayRetryBaseMs 1500 · keepAliveTimeoutMs 5000 · clientErrorBackstopMs 30000 · exposureLogIntervalMs 3600000 · streamKeepAliveMs 15000 · preflightHoldMs 30000 · preflightDeadEndCooldownMs 300000 · proxyKeepAliveMaxMs 55000 · throttle429Enabled true · throttle429MaxConsecutive 5 · throttle429BaseDelayMs 2000 · throttle429MaxDelayMs 30000.", + "tt.network": "network — transport timeouts, replays & the 429 egress governor: upstreamTimeoutMs 720000 · requestWatchdogMs 2× · replayRetryMax 3 · replayRetryBaseMs 1500 · keepAliveTimeoutMs 5000 · clientErrorBackstopMs 30000 · exposureLogIntervalMs 3600000 · streamKeepaliveMs 15000 · preflightHoldMs 30000 · preflightDeadEndCooldownMs 300000 · proxyKeepAliveMaxMs 55000 · throttle429Enabled true · throttle429MaxConsecutive 5 · throttle429BaseDelayMs 2000 · throttle429MaxDelayMs 30000 · throttle429Scope credential.", "tt.guardrails": "guardrails — pi/omp plugin tool-output + stream text-run guardrails: bashTimeout 60 · maxOutputBytes 50000 · repetition true · textRunThreshold 200.", "tt.persist": "persist — session persistence to disk: enabled true · zstd false · debounceMs 500 · tailTokens 16384 · epermAlertThreshold 5 · epermAlertRepeatMs 0.", "tt.storage": "storage — unified storage sidecar: unified legacy — file tier selects the lane-wide storage mode; env BILI_STORAGE_UNIFIED wins when set.", @@ -386,7 +386,7 @@ "tt.ccx": "codexCompact —— intercept | pass(默认 intercept):在 bili 内处理 Codex 压缩请求或转给上游 —— 显式形态 + 以普通请求形态过线的内联自动压缩(prompt 文本检测,#3008;自定义 compact_prompt 按原样转发)。", "ttl.ttl": "ccrRetrievalTtlMs —— CCR 回取指针有效期(默认 600000;0 禁用回取)。", "ttl.tmp": "decompressTmpCap —— 临时展开块的大小上限(默认 50)。", - "tt.network": "network —— 传输超时、重放与 429 出站治理:upstreamTimeoutMs 720000 · requestWatchdogMs 2× · replayRetryMax 3 · replayRetryBaseMs 1500 · keepAliveTimeoutMs 5000 · clientErrorBackstopMs 30000 · exposureLogIntervalMs 3600000 · streamKeepAliveMs 15000 · preflightHoldMs 30000 · preflightDeadEndCooldownMs 300000 · proxyKeepAliveMaxMs 55000 · throttle429Enabled true · throttle429MaxConsecutive 5 · throttle429BaseDelayMs 2000 · throttle429MaxDelayMs 30000。", + "tt.network": "network —— 传输超时、重放与 429 出站治理:upstreamTimeoutMs 720000 · requestWatchdogMs 2× · replayRetryMax 3 · replayRetryBaseMs 1500 · keepAliveTimeoutMs 5000 · clientErrorBackstopMs 30000 · exposureLogIntervalMs 3600000 · streamKeepAliveMs 15000 · preflightHoldMs 30000 · preflightDeadEndCooldownMs 300000 · proxyKeepAliveMaxMs 55000 · throttle429Enabled true · throttle429MaxConsecutive 5 · throttle429BaseDelayMs 2000 · throttle429MaxDelayMs 30000 · throttle429Scope credential。", "tt.guardrails": "guardrails —— pi/omp 插件工具输出与流式 text-run 守卫:bashTimeout 60 · maxOutputBytes 50000 · repetition true · textRunThreshold 200。", "tt.persist": "persist —— 会话落盘持久化:enabled true · zstd false · debounceMs 500 · tailTokens 16384 · epermAlertThreshold 5 · epermAlertRepeatMs 0。", "tt.storage": "storage —— 统一存储边车:unified legacy —— 文件层选定全 lane 存储模式;设置了 BILI_STORAGE_UNIFIED 时环境变量优先。",