Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CONFIGURATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -171,6 +171,7 @@ This index is generated from `website/config-reference/*.yaml` — edit the seed
| `network.throttle429MaxConsecutive` | number | 5 | BILI_THROTTLE_429_MAX_CONSECUTIVE | Consecutive upstream 429s before the hold turns terminal (local 424 outside every known SDK retry set); one probe per backoff window keeps checking until any non-429 response clears an account hold, or the matching model's non-429 response clears its model hold. |
| `network.throttle429BaseDelayMs` | number | 2000 | BILI_THROTTLE_429_BASE_DELAY_MS | Base backoff for the 429 hold (doubles per consecutive 429, jittered into [0.5×, 1×], capped by throttle429MaxDelayMs; an upstream Retry-After wins when larger). |
| `network.throttle429MaxDelayMs` | number | 30000 | BILI_THROTTLE_429_MAX_DELAY_MS | Ceiling for the 429 hold backoff and for the terminal-phase probe interval. |
| `network.throttle429Scope` | string | credential | BILI_THROTTLE_429_SCOPE | Hold granularity for marker-less upstream 429s (#5199). `credential` (default) keeps #5696's conservative semantics — any 429 without a recognized model marker (`keyring_model_temporarily_unavailable`, or a bili relay's `bili_throttle_hold` scope=model) holds the whole origin+credential, so every model behind that credential waits out the backoff. `model` keys the hold by the send's request model instead, for gateways that front many independently-limited models behind one credential (LiteLLM/one-api/opencode.ai-style relays where one model's RPM exhaustion must not freeze its siblings); requests whose model cannot be identified remain credential-wide, and recognized markers keep their own semantics in both modes. |
| `network.maxShrinkPerCompress` | number | unset | BILI_MAX_SHRINK_PER_COMPRESS | Cap on how much one compression may shrink history (anti-collapse guard). |
| `network.proxyKeepAliveMaxMs` | number | 55000 (0 = one-shot connections) | BILI_PROXY_KEEPALIVE_MAX_MS | Max lifetime of a connection through an outbound egress proxy before forcing reconnection (avoids stale NAT mappings). |
| `network.postResponseLingerMs` | number | 5000 | BILI_POST_RESPONSE_LINGER_MS | Graceful-close budget for proxy-initiated closes after the response completed: the socket is held waiting for the peer's FIN/TLS close_notify and destroyed at the deadline (reason=linger-backstop). |
Expand Down Expand Up @@ -1980,6 +1981,7 @@ File keys resolve only when the matching env var is unset. Defaults in parenthes
| `BILI_THROTTLE_429_BASE_DELAY_MS` | `network.throttle429BaseDelayMs` | 2000 |
| `BILI_THROTTLE_429_MAX_CONSECUTIVE` | `network.throttle429MaxConsecutive` | 5 |
| `BILI_THROTTLE_429_MAX_DELAY_MS` | `network.throttle429MaxDelayMs` | 30000 |
| `BILI_THROTTLE_429_SCOPE` | `network.throttle429Scope` | credential |
| `BILI_UPDATE_CHECK_INTERVAL_MS` | `update.checkIntervalMs` | 180000 |
| `BILI_UPDATE_POLICY_URL` | `update.policyUrl` | official repo raw URL |
| `BILI_UPDATE_REGISTRY` | `update.registry` | "npmjs" (registry.npmjs.org) |
Expand Down
2 changes: 2 additions & 0 deletions CONFIGURATION.zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -171,6 +171,7 @@
| `network.throttle429MaxConsecutive` | number | 5 | BILI_THROTTLE_429_MAX_CONSECUTIVE | 连续上游 429 达到该次数后 hold 转为终局(本地 424,不在任何已知 SDK 重试集合内);每个退避窗口仍放行一次探测,直到账号级 hold 收到任意非 429 响应,或模型级 hold 收到同一模型的非 429 响应后清除。 |
| `network.throttle429BaseDelayMs` | number | 2000 | BILI_THROTTLE_429_BASE_DELAY_MS | 429 hold 的基础退避(每连续一次 429 翻倍、抖动到 [0.5×, 1×]、以 throttle429MaxDelayMs 封顶;上游 Retry-After 更大时优先)。 |
| `network.throttle429MaxDelayMs` | number | 30000 | BILI_THROTTLE_429_MAX_DELAY_MS | 429 hold 退避的上限,也是终局阶段探测间隔的上限。 |
| `network.throttle429Scope` | string | credential | BILI_THROTTLE_429_SCOPE | 无标记上游 429 的 hold 粒度(#5199)。`credential`(默认)保持 #5696 的保守语义——凡未携带可识别模型标记(`keyring_model_temporarily_unavailable`,或 bili 中继的 `bili_throttle_hold` scope=model)的 429 均按整个 origin+凭据持有,该凭据下所有模型一起退避等待。`model` 改按发送请求的模型分桶持有,适用于单凭据背后多模型独立限流的网关(LiteLLM/one-api/opencode.ai 类中继,一个模型的 RPM 耗尽不应冻结兄弟模型);无法识别模型的请求仍按凭据级持有,两种模式下已识别标记的语义均不变。 |
| `network.maxShrinkPerCompress` | number | unset | BILI_MAX_SHRINK_PER_COMPRESS | 单次压缩允许收缩历史的上限(防塌缩护栏)。 |
| `network.proxyKeepAliveMaxMs` | number | 55000 (0 = one-shot connections) | BILI_PROXY_KEEPALIVE_MAX_MS | 经出站代理的连接最大存活时长,到期强制重建(规避陈旧 NAT 映射)。 |
| `network.postResponseLingerMs` | number | 5000 | BILI_POST_RESPONSE_LINGER_MS | 代理主动发起的响应后关闭的优雅关闭预算:socket 挂起等待对端 FIN/TLS close_notify,到点销毁(reason=linger-backstop)。 |
Expand Down Expand Up @@ -1957,6 +1958,7 @@ ACP 原生 agent(当前为 `pi` 扩展)会在每个进程内向代理上报
| `BILI_THROTTLE_429_BASE_DELAY_MS` | `network.throttle429BaseDelayMs` | 2000 |
| `BILI_THROTTLE_429_MAX_CONSECUTIVE` | `network.throttle429MaxConsecutive` | 5 |
| `BILI_THROTTLE_429_MAX_DELAY_MS` | `network.throttle429MaxDelayMs` | 30000 |
| `BILI_THROTTLE_429_SCOPE` | `network.throttle429Scope` | credential |
| `BILI_UPDATE_CHECK_INTERVAL_MS` | `update.checkIntervalMs` | 180000 |
| `BILI_UPDATE_POLICY_URL` | `update.policyUrl` | official repo raw URL |
| `BILI_UPDATE_REGISTRY` | `update.registry` | "npmjs" (registry.npmjs.org) |
Expand Down
2 changes: 1 addition & 1 deletion LIFECYCLE.md
Original file line number Diff line number Diff line change
Expand Up @@ -96,7 +96,7 @@
| `BILI_EXPOSURE_LOG_INTERVAL_MS` | 3600000(0 禁用,垃圾值回落 3600000) |
| `BILI_MAX_SHRINK_PER_COMPRESS` | unset(不做收缩引导) |
| `BILI_REPLAY_RETRY_MAX` / `BILI_REPLAY_RETRY_BASE_MS` | 3 / 1500 |
| `BILI_THROTTLE_429` / `BILI_THROTTLE_429_MAX_CONSECUTIVE` / `BILI_THROTTLE_429_BASE_DELAY_MS` / `BILI_THROTTLE_429_MAX_DELAY_MS` | true / 5 / 2000 / 30000(#4204 出站 429 治理器:同一 origin+凭据连续上游 429 后本地 hold、零出站,退避×抖动、尊重 Retry-After;达上限转终局 424(不在各 SDK 重试集合内),每窗口一次探测自动恢复,任何非 429 响应即清除;`false` 恢复旧的总是转发行为) |
| `BILI_THROTTLE_429` / `BILI_THROTTLE_429_MAX_CONSECUTIVE` / `BILI_THROTTLE_429_BASE_DELAY_MS` / `BILI_THROTTLE_429_MAX_DELAY_MS` / `BILI_THROTTLE_429_SCOPE` | true / 5 / 2000 / 30000 / credential(#4204 出站 429 治理器:同一 origin+凭据连续上游 429 后本地 hold、零出站,退避×抖动、尊重 Retry-After;达上限转终局 424(不在各 SDK 重试集合内),每窗口一次探测自动恢复,任何非 429 响应即清除;`false` 恢复旧的总是转发行为。#5199:scope=`model` 时无标记 429 按请求模型分桶 hold——单凭据多模型独立限流网关(LiteLLM/one-api 类)一个模型的 RPM 耗尽不再冻结兄弟模型;无法识别模型的请求仍按凭据级) |
| `BILI_STREAM_KEEPALIVE_MS` | 15000(0 禁用 hold) |
| `BILI_PREFLIGHT_HOLD_MS` | 30000 |
| `BILI_PREFLIGHT_DEAD_END_COOLDOWN_MS` | 300000 |
Expand Down
7 changes: 7 additions & 0 deletions src/config.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2264,6 +2264,13 @@ type FileConfig = {
/** 429-hold backoff ceiling in ms; also the probe cadence once the hold is terminal (#4204;
* was env-only BILI_THROTTLE_429_MAX_DELAY_MS; default 30000). */
throttle429MaxDelayMs?: number;
/** Hold granularity for marker-less upstream 429s (#5199; env BILI_THROTTLE_429_SCOPE; default "credential").
* "credential" — #5696's conservative default: any 429 without a recognized model marker holds the whole
* origin+credential (every model, every session on it). "model" — hold only the send's request model, for
* gateways fronting many independently-limited models behind one credential (LiteLLM/one-api/opencode.ai-style
* relays); requests without an identifiable model remain credential-wide, and recognized KeyRing markers keep
* their own semantics in both modes. */
throttle429Scope?: "credential" | "model";
/** Per-compress shrink steering factor in (0,1] (was BILI_MAX_SHRINK_PER_COMPRESS; default unset). */
maxShrinkPerCompress?: number;
/** Outbound proxy keep-alive ceiling, 0 for one-shot (was BILI_PROXY_KEEPALIVE_MAX_MS; default 55000). */
Expand Down
4 changes: 2 additions & 2 deletions src/degenerate-retry.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
import { fetchWithTimeout } from "./fetch-util.js";
import { beginProvisionalUpstreamThrottle429, keyringModel429Scope, noteUpstreamThrottleResolved, releaseUpstreamThrottleProbe, requestModelFromBody, reserveUpstreamThrottleClassification, upstreamThrottleGate, type ThrottleClassificationReservation } from "./upstream-throttle.js";
import { beginProvisionalUpstreamThrottle429, noteUpstreamThrottleResolved, releaseUpstreamThrottleProbe, requestModelFromBody, reserveUpstreamThrottleClassification, throttle429ModelScope, upstreamThrottleGate, type ThrottleClassificationReservation } from "./upstream-throttle.js";
import { hasCompactionTrigger } from "./codex-compact.js";
import { proxyDispatcher } from "./upstream-proxy.js";
import type { WireProtocol } from "./util.js";
Expand Down Expand Up @@ -191,7 +191,7 @@ export function makeContinuationRefetch(opts: ContinuationRetryOpts): () => Prom
// is the continuation callback's 250ms latency budget.
const bodyRead = readSmallThrottleErrorBody(r.response)
.then((errorBody) => {
const modelScope = keyringModel429Scope(r.response.status, requestModel, errorBody);
const modelScope = throttle429ModelScope(r.response.status, requestModel, errorBody);
settle(modelScope);
})
.catch(() => {
Expand Down
17 changes: 17 additions & 0 deletions src/knobs.ts
Original file line number Diff line number Diff line change
Expand Up @@ -145,6 +145,23 @@ export function throttle429MaxDelayMs(): number {
return Number.isFinite(v) && v >= 0 ? v : THROTTLE_429_MAX_DELAY_MS;
}

/** BILI_THROTTLE_429_SCOPE > network.throttle429Scope > "credential". Hold
* granularity for marker-less upstream 429s (#5199/#5853 decision 1, option
* A): "credential" keeps #5696's conservative default — an unrecognized 429
* body (RPM/quota text, opaque relay rewrites) holds the whole
* origin+credential, freezing every model behind it; "model" keys that hold
* by the send's request model, for gateways that front many independently
* limited models behind one credential (LiteLLM/one-api/opencode.ai-style
* relays). Recognized KeyRing/bili-relay markers keep their own semantics in
* both modes; requests whose model cannot be identified stay
* credential-wide, so worst-case protection never widens beyond legacy.
* Read per settlement so the switch flips live. */
export function throttle429Scope(): "credential" | "model" {
const env = process.env.BILI_THROTTLE_429_SCOPE;
const raw = env !== undefined && env !== "" ? env : fileNetwork().throttle429Scope;
return raw === "model" ? "model" : "credential";
}

/** BILI_MAX_SHRINK_PER_COMPRESS > network.maxShrinkPerCompress > unset (no steering). */
export function maxShrinkPerCompress(): number | undefined {
const v = tNum("BILI_MAX_SHRINK_PER_COMPRESS", fileNetwork().maxShrinkPerCompress);
Expand Down
6 changes: 3 additions & 3 deletions src/loop/core.ts
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ import { applyConfiguredCompression } from "../external-summary-compress.js";
import { executeSearchContextTarget, resolveDecompress } from "../decompress-shared.js";
import { toolFail, type ProxyToolResult } from "../proxy-tool-result.js";
import { fetchWithRetry, UpstreamHttpError } from "../fetch-util.js";
import { keyringModel429Scope, noteUpstreamThrottle429, noteUpstreamThrottleResolved } from "../upstream-throttle.js";
import { noteUpstreamThrottle429, noteUpstreamThrottleResolved, throttle429ModelScope } from "../upstream-throttle.js";
import { classifyUpstreamFailure, type SkippedProxyInfo, type UpstreamFailureKind } from "../upstream-fail.js";
import { formatUpstreamError, proxyDispatcher } from "../upstream-proxy.js";
import { warnCacheCollapse } from "../cache-warn.js";
Expand Down Expand Up @@ -601,7 +601,7 @@ export async function* runCompressLoop(
signal,
(info) => {
if (info.status === 429) {
const modelScope = keyringModel429Scope(info.status, requestModel, info.detail);
const modelScope = throttle429ModelScope(info.status, requestModel, info.detail);
noteUpstreamThrottle429(requestOptions.url, requestOptions.headers, null, Date.now(), modelScope); // Retry-After is not carried on replay metadata.
}
// #189: correlate the rejection with the rewrite that
Expand All @@ -622,7 +622,7 @@ export async function* runCompressLoop(
return upstreamSend;
} catch (error) {
if (error instanceof UpstreamHttpError && error.status === 429) {
const modelScope = keyringModel429Scope(error.status, requestModel, error.body);
const modelScope = throttle429ModelScope(error.status, requestModel, error.body);
noteUpstreamThrottle429(requestOptions.url, requestOptions.headers, null, Date.now(), modelScope);
}
throw error;
Expand Down
4 changes: 2 additions & 2 deletions src/server/relay.ts
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ import { safePrefix, safeSuffix, scrubLoneSurrogatesOnWire } from "../text-safe.
import { clearUpstreamAlertsForHost, noteCircuitFailFastLog, recordUpstreamAlert, recordUpstreamCircuitFailure, recordUpstreamCircuitSuccess, upstreamCircuitBlockReason, upstreamCircuitLastFailureKind, upstreamCircuitRetryAfterSeconds } from "../upstream-alerts.js";
import { describeUpstreamFailure, formatUpstreamError, proxyDispatcher, recordUpstreamConnection } from "../upstream-proxy.js";
import { markExpectedFailFast, withUpstreamFailure, type UpstreamFailureKind } from "../upstream-fail.js";
import { keyringModel429Scope, noteUpstreamThrottle429, noteUpstreamThrottleResolved, releaseUpstreamThrottleProbe, requestModelFromBody, type ThrottleLocalReject, upstreamThrottleGate, upstreamThrottlePeekReject } from "../upstream-throttle.js";
import { noteUpstreamThrottle429, noteUpstreamThrottleResolved, releaseUpstreamThrottleProbe, requestModelFromBody, throttle429ModelScope, type ThrottleLocalReject, upstreamThrottleGate, upstreamThrottlePeekReject } from "../upstream-throttle.js";
import { applyEstimateCalibration, currentCalibrationFactor, inspectContextOverflow, isMarkerlessOverflowReject, MIN_ASSUMED_WINDOW_TOKENS, normalizeUpstreamOrigin, usageOutputTotal, usageTotals, type ContextOverflowInfo, type WireProtocol } from "../util.js";
import { billOutputUsage } from "../billing.js";
import { DUMP_PLAINTEXT_BANNER } from "../dump-banner.js";
Expand Down Expand Up @@ -1691,7 +1691,7 @@ async function forwardCaptured(
// keyring_all_limited and every ambiguous response remain account-wide.
if (req.method !== "GET" && req.method !== "HEAD") {
if (upstream.status === 429) {
const modelScope = keyringModel429Scope(upstream.status, throttleModel, errBody);
const modelScope = throttle429ModelScope(upstream.status, throttleModel, errBody);
noteUpstreamThrottle429(upstreamUrl, headers, upstream.headers.get("retry-after"), Date.now(), modelScope, throttleProbeToken);
} else {
noteUpstreamThrottleResolved(upstreamUrl, headers, upstream.status, Date.now(), throttleModel, throttleProbeToken);
Expand Down
22 changes: 22 additions & 0 deletions src/upstream-throttle.ts
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@ import {
throttle429Enabled,
throttle429MaxConsecutive,
throttle429MaxDelayMs,
throttle429Scope,
} from "./knobs.js";

/** Bounded state table (same posture as MAX_CIRCUITS in upstream-alerts.ts). */
Expand Down Expand Up @@ -241,6 +242,27 @@ export function keyringModel429Scope(
return undefined;
}

/**
* The hold scope for an observed 429, combining #5696's marker discipline with
* the #5199/#5853 knob: a recognized KeyRing / bili-relay marker wins exactly
* as before; otherwise, when network.throttle429Scope is "model" and the send
* carried an identifiable request model, a marker-less gateway 429 (RPM/quota
* text, opaque relay-rewritten bodies — LiteLLM/one-api/opencode.ai shapes
* where one model's pool exhaustion must not freeze its siblings) settles
* model-scoped instead of credential-wide. Model-less sends keep the legacy
* credential key, so worst-case protection never widens beyond legacy.
*/
export function throttle429ModelScope(
status: number,
requestModel: string | undefined,
responseBody: string | Uint8Array | null | undefined,
): string | undefined {
const recognized = keyringModel429Scope(status, requestModel, responseBody);
if (recognized !== undefined) return recognized;
if (status !== 429 || throttle429Scope() !== "model") return undefined;
return requestModel !== undefined && requestModel.length > 0 ? requestModel : undefined;
}

function modelHoldKey(key: string, model: string): string {
return `${key}\u0000model\u0000${model}`;
}
Expand Down
Loading
Loading