From e248090ff52af8c772c51481022e5eb32520e84f Mon Sep 17 00:00:00 2001 From: Aswin-Ram-K <185140014+Aswin-Ram-K@users.noreply.github.com> Date: Wed, 4 Nov 2026 15:23:10 -0600 Subject: [PATCH 1/3] fix: isolate KeyRing model rate limit holds --- CHANGELOG.md | 1 + CONFIGURATION.md | 4 +- CONFIGURATION.zh-CN.md | 4 +- src/acp-status.ts | 11 +- src/degenerate-retry.ts | 135 ++- src/egress-shed.ts | 7 +- src/loop/core.ts | 15 +- src/server/handle.ts | 43 +- src/server/relay.ts | 70 +- src/upstream-throttle.ts | 773 ++++++++++++++++-- tests/issue4204-throttle-governor.test.ts | 455 ++++++++++- tests/issue4725-loop-429-observe.test.ts | 12 +- .../issue4743-degenerate-refetch-429.test.ts | 420 +++++++++- tests/proxy-noinject-tag-strip.test.ts | 143 +++- tests/upstream-error-log.test.ts | 519 +++++++++++- 15 files changed, 2431 insertions(+), 181 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 971f00118d..d13a74e0ab 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,6 +10,7 @@ Versions follow the merge of a `*_release-v*` branch; CI publishes to npm on tag - **The full-resend watermark fastpath (#5403/#5418) is now ON by default (GA)** after its §7.4 human review: plugin-lane chat-completions turns whose prefix is byte-stable skip the full history re-parse/re-pin on hit (the #5403 OOM producer), any doubt still falls back to the untouched full-parse path per turn. Opt out per install with `rollout.features.pluginFullResendWatermark: false` (or a percent) or `BILI_ROLLOUT_PLUGIN_FULL_RESEND_WATERMARK=0`. ### Fixed +- **KeyRing model-local 429s no longer throttle other models on the same credential.** Bili gives an independent origin+credential+model hold only to HTTP 429 responses whose structured KeyRing or Bili marker has an exact request-model match; `keyring_all_limited`, direct OpenCode monthly quotas, malformed bodies, and mismatches remain origin+credential holds. Large passthrough bodies use a top-level model scan, while Gemini/Vertex/v1 and side-lane paths carry their actual wire model. Account/model probe slots are request-owned and canceled sends release only their token; concurrent provisional 429 observations settle independently. Continuation retries reserve bounded classification capacity per origin+credential before egress; saturation skips only the optional refetch, without a global gate or discarding account-wide evidence. Continuation reads retain the upstream idle/abort policy. Side-lane SSE retains its stream mode so render-tag stripping still runs. Regressions: `tests/issue4204-throttle-governor.test.ts`, `tests/issue4743-degenerate-refetch-429.test.ts`, `tests/proxy-noinject-tag-strip.test.ts`, and `tests/upstream-error-log.test.ts`. - **External-summary mode no longer reports protected or already-folded ranges as failed compression.** An unchanged kernel preview now returns the normal idempotent no-op or kernel-error receipt without a paid summarizer call; validated internal probe summaries prevent an irrelevant submitted-hint length from blocking that receipt. Probe-created blocks with no recoverable content still refuse rather than folding an unreviewed hint. Regression: `tests/external-summary-runtime.test.ts` (24/24). * **#5412: 异步存储后端的「压缩冻结会话」现在也能自愈。** #5408 事故(一个会话连续 17 次摘要漂移后被保护机制冻住、8 小时无人知晓)的主修复 #5409 只教了同步存储路径(SQLite 直写)自救:漂移挂起期间保存会自动升级为全量重导入并清掉索引里的残留行。异步路径(opt-in 网络后端,如 mysql)那份孪生代码没搬——同样被冻住的会话在那上面永远救不回来,每次保存都重写不完整视图、下轮采样继续漂移。本 PR 在 `src/storage/shadow.ts` 的异步入队路径补上同一行对齐(`completeLog = hasRawTail || driftedSessions.has(id)`),行为与同步路径逐字一致;回归测试同时钉住正门(latch 挂起的折叠保存清掉外来行)与负门(未挂起保存不清扫)。 diff --git a/CONFIGURATION.md b/CONFIGURATION.md index fe422b9461..5f68c9e3d9 100644 --- a/CONFIGURATION.md +++ b/CONFIGURATION.md @@ -166,8 +166,8 @@ This index is generated from `website/config-reference/*.yaml` — edit the seed | `network.preflightDeadEndCooldownMs` | number | 300000 | BILI_PREFLIGHT_DEAD_END_COOLDOWN_MS | Cooldown after a dead-end preflight before retrying reconciliation for the same session. | | `network.replayRetryMax` | number | 3 (1 disables replays) | BILI_REPLAY_RETRY_MAX | Maximum transport-failure replays of an identical request before surfacing the error. | | `network.replayRetryBaseMs` | number | 1500 (0 disables the delay) | BILI_REPLAY_RETRY_BASE_MS | Base backoff delay between replays (exponential). | -| `network.throttle429Enabled` | boolean | true | BILI_THROTTLE_429 | Egress-side 429 governor (#4204): after consecutive upstream 429s, further sends to that origin+credential are answered locally with zero egress instead of being forwarded; set false to restore the legacy always-forward behavior. | -| `network.throttle429MaxConsecutive` | number | 5 | BILI_THROTTLE_429_MAX_CONSECUTIVE | Consecutive upstream 429s before the hold turns terminal (local 424 outside every known SDK retry set); one probe per backoff window keeps checking until any non-429 response clears it. | +| `network.throttle429Enabled` | boolean | true | BILI_THROTTLE_429 | Egress-side 429 governor (#4204): repeated upstream 429s are answered locally instead of forwarded. Account limits and unrecognized/malformed responses hold origin+credential; only KeyRing's exact `keyring_model_temporarily_unavailable` code with a matching request model holds origin+credential+model, leaving other models eligible. A mismatch or missing model remains account-wide; false restores the legacy always-forward behavior. | +| `network.throttle429MaxConsecutive` | number | 5 | BILI_THROTTLE_429_MAX_CONSECUTIVE | Consecutive upstream 429s before the hold turns terminal (local 424 outside every known SDK retry set); one probe per backoff window keeps checking until any non-429 response clears an account hold, or the matching model's non-429 response clears its model hold. | | `network.throttle429BaseDelayMs` | number | 2000 | BILI_THROTTLE_429_BASE_DELAY_MS | Base backoff for the 429 hold (doubles per consecutive 429, jittered into [0.5×, 1×], capped by throttle429MaxDelayMs; an upstream Retry-After wins when larger). | | `network.throttle429MaxDelayMs` | number | 30000 | BILI_THROTTLE_429_MAX_DELAY_MS | Ceiling for the 429 hold backoff and for the terminal-phase probe interval. | | `network.maxShrinkPerCompress` | number | unset | BILI_MAX_SHRINK_PER_COMPRESS | Cap on how much one compression may shrink history (anti-collapse guard). | diff --git a/CONFIGURATION.zh-CN.md b/CONFIGURATION.zh-CN.md index 116ca6fe51..196e9dfa99 100644 --- a/CONFIGURATION.zh-CN.md +++ b/CONFIGURATION.zh-CN.md @@ -166,8 +166,8 @@ | `network.preflightDeadEndCooldownMs` | number | 300000 | BILI_PREFLIGHT_DEAD_END_COOLDOWN_MS | 死胡同 preflight 后,对同一会话重试校准的冷却时长。 | | `network.replayRetryMax` | number | 3 (1 disables replays) | BILI_REPLAY_RETRY_MAX | 同一请求在向上游暴露错误前的最大传输失败重放次数。 | | `network.replayRetryBaseMs` | number | 1500 (0 disables the delay) | BILI_REPLAY_RETRY_BASE_MS | 重放之间的基础退避延迟(指数增长)。 | -| `network.throttle429Enabled` | boolean | true | BILI_THROTTLE_429 | 出站 429 治理器(#4204):同一 origin+凭据连续收到上游 429 后,后续请求在本地应答(零出站)而不再转发;设为 false 恢复旧的总是转发行为。 | -| `network.throttle429MaxConsecutive` | number | 5 | BILI_THROTTLE_429_MAX_CONSECUTIVE | 连续上游 429 达到该次数后 hold 转为终局(本地 424,不在任何已知 SDK 重试集合内);每个退避窗口仍放行一次探测,直到出现非 429 响应清除。 | +| `network.throttle429Enabled` | boolean | true | BILI_THROTTLE_429 | 出站 429 治理器(#4204):连续上游 429 会在本地应答而不再转发。账号限额及无法识别/格式错误的响应按 origin+凭据持有;只有 KeyRing 精确返回 `keyring_model_temporarily_unavailable` 且响应模型与请求模型完全相同时,才按 origin+凭据+模型持有,让其他模型继续可用。模型不匹配或缺失仍按账号级处理;设为 false 恢复旧的总是转发行为。 | +| `network.throttle429MaxConsecutive` | number | 5 | BILI_THROTTLE_429_MAX_CONSECUTIVE | 连续上游 429 达到该次数后 hold 转为终局(本地 424,不在任何已知 SDK 重试集合内);每个退避窗口仍放行一次探测,直到账号级 hold 收到任意非 429 响应,或模型级 hold 收到同一模型的非 429 响应后清除。 | | `network.throttle429BaseDelayMs` | number | 2000 | BILI_THROTTLE_429_BASE_DELAY_MS | 429 hold 的基础退避(每连续一次 429 翻倍、抖动到 [0.5×, 1×]、以 throttle429MaxDelayMs 封顶;上游 Retry-After 更大时优先)。 | | `network.throttle429MaxDelayMs` | number | 30000 | BILI_THROTTLE_429_MAX_DELAY_MS | 429 hold 退避的上限,也是终局阶段探测间隔的上限。 | | `network.maxShrinkPerCompress` | number | unset | BILI_MAX_SHRINK_PER_COMPRESS | 单次压缩允许收缩历史的上限(防塌缩护栏)。 | diff --git a/src/acp-status.ts b/src/acp-status.ts index 6d25d84717..8871f780d0 100644 --- a/src/acp-status.ts +++ b/src/acp-status.ts @@ -399,17 +399,16 @@ export function handleAcpStatus(args: Record, ctx: AcpStatusCtx } if (openCircuits.length > 4) extra.push(`UPSTREAM CIRCUIT OPEN — (+${openCircuits.length - 4} more host(s))`); } - // #4204: display counterpart of the egress-side 429 governor — holds are - // per origin+credential and answer locally (zero egress) while armed, so - // hosts/models can see WHY sends are held back and when the next probe - // fires. Display-only; no wire decision lives here. + // #4204: display counterpart of the egress-side 429 governor. Account + // holds are per origin+credential; a verified KeyRing model refusal has + // its own origin+credential+model hold. Both answer locally (zero egress). const throttleHolds = activeThrottleHolds(); if (throttleHolds.length > 0) { for (const h of throttleHolds.slice(0, 4)) { extra.push(""); - extra.push(`UPSTREAM 429 HOLD — ${h.host}: ${h.streak} consecutive upstream 429(s); outbound sends answered locally until ${new Date(h.nextAttemptAt).toISOString()} (${h.terminal ? "TERMINAL — local 424 outside SDK retry sets until a probe succeeds" : "backoff window"}); zero egress while held, clears on the next non-429 response (#4204).`); + extra.push(`UPSTREAM 429 HOLD — ${h.host}${h.model === undefined ? "" : ` (model ${h.model})`}: ${h.streak} consecutive upstream 429(s); outbound sends answered locally until ${new Date(h.nextAttemptAt).toISOString()} (${h.terminal ? "TERMINAL — local 424 outside SDK retry sets until a probe succeeds" : "backoff window"}); zero egress while held, clears on a resolved non-429 response in the same scope (#4204).`); } - if (throttleHolds.length > 4) extra.push(`UPSTREAM 429 HOLD — (+${throttleHolds.length - 4} more origin+credential key(s))`); + if (throttleHolds.length > 4) extra.push(`UPSTREAM 429 HOLD — (+${throttleHolds.length - 4} more origin+credential[+model] key(s))`); } const shrinkAt = ctx.session.stats.emergencyShrinkAt; if (typeof shrinkAt === "number") { diff --git a/src/degenerate-retry.ts b/src/degenerate-retry.ts index a21b28bd73..ab09d676ad 100644 --- a/src/degenerate-retry.ts +++ b/src/degenerate-retry.ts @@ -1,10 +1,29 @@ import { fetchWithTimeout } from "./fetch-util.js"; -import { noteUpstreamThrottle429, noteUpstreamThrottleResolved, upstreamThrottleLocalReject } from "./upstream-throttle.js"; +import { beginProvisionalUpstreamThrottle429, keyringModel429Scope, noteUpstreamThrottleResolved, releaseUpstreamThrottleProbe, requestModelFromBody, reserveUpstreamThrottleClassification, upstreamThrottleGate, type ThrottleClassificationReservation } from "./upstream-throttle.js"; import { hasCompactionTrigger } from "./codex-compact.js"; import { proxyDispatcher } from "./upstream-proxy.js"; import type { WireProtocol } from "./util.js"; import { appendTrailingUserText } from "./wire-body.js"; +const MAX_THROTTLE_ERROR_BODY_BYTES = 16 * 1024; +// Keep callback latency separate from the bounded body read used for scope refinement. +const MAX_THROTTLE_ERROR_READ_MS = 250; + +function cancelThrottleErrorReader(reader: ReadableStreamDefaultReader): void { + try { + void reader.cancel().catch(() => undefined); + } catch { + /* cancellation is best-effort; never make the client wait for it */ + } +} + +function releaseThrottleErrorReader(reader: ReadableStreamDefaultReader): void { + try { + reader.releaseLock(); + } catch { + /* an in-flight read still owns the lock */ + } +} // #732/#821 ran this retry for the compress loop, which owns the request body // AND the response framing. The plugin pipe (plugin mode, and proxy-mode chat // SSE that skipped compress injection) owns neither: it forwards the agent's @@ -34,6 +53,8 @@ interface ContinuationRetryOpts { protocol: WireProtocol; /** The body that was forwarded upstream (the agent's own, in plugin mode). */ body: string | Buffer; + /** Exact request model; required to isolate a matching KeyRing model refusal. */ + model?: string; upstreamUrl: string; reqHeaders: Record; proxyUrl?: string; @@ -51,6 +72,36 @@ interface ContinuationRetryOpts { wrapBody?: (body: string | Buffer) => string | Buffer; } +async function readSmallThrottleErrorBody(response: Response): Promise { + const reader = response.body?.getReader(); + if (!reader) return undefined; + const chunks: Uint8Array[] = []; + let bytes = 0; + try { + while (true) { + const result = await reader.read(); + if (result.done) { + const body = new Uint8Array(bytes); + let offset = 0; + for (const chunk of chunks) { + body.set(chunk, offset); + offset += chunk.byteLength; + } + return body; + } + bytes += result.value.byteLength; + if (bytes > MAX_THROTTLE_ERROR_BODY_BYTES) { + cancelThrottleErrorReader(reader); + return undefined; + } + chunks.push(result.value); + } + } catch { + return undefined; + } finally { + releaseThrottleErrorReader(reader); + } +} /** Build the re-request used when a turn ends with no visible output. Returns a * function for the stream pipe: it yields the retry's response body, or null * when the retry cannot be issued (unbuildable body, upstream error, client @@ -81,16 +132,31 @@ export function makeContinuationRefetch(opts: ContinuationRetryOpts): () => Prom return null; } const wireRetryBody = opts.wrapBody ? opts.wrapBody(retryBody) : retryBody; + const requestModel = opts.model ?? requestModelFromBody(retryBody); + let probeToken: symbol | undefined; + let classificationReservation: ThrottleClassificationReservation | undefined; + let settleProvisional: ((modelScope?: string) => void) | undefined; try { + classificationReservation = reserveUpstreamThrottleClassification(opts.upstreamUrl, opts.reqHeaders); + if (classificationReservation === undefined) { + opts.log("info", `[${opts.label}] [plugin] degenerate-terminal retry skipped: throttle classification capacity is full`); + return null; + } opts.resign?.(opts.reqHeaders, wireRetryBody); - // #4743: the retry is an egress send like any other — feed the 429 - // governor (observe only, no local gate: the retry is bounded by - // the degenerate-turn ladder, and holding IT locally would strand - // the client's already-open response stream). A 429 verdict arms - // the hold for the NEXT main-lane send; any other resolved status - // clears it — same one-observation-per-request discipline as relay. - const throttleReject = upstreamThrottleLocalReject(opts.upstreamUrl, opts.reqHeaders); + // #4743: the retry is an egress send like any other. Its local + // admission gate returns a token so only this send can release the + // probe slots it claimed. + const throttleGate = upstreamThrottleGate( + opts.upstreamUrl, + opts.reqHeaders, + Date.now(), + requestModel, + ); + probeToken = throttleGate.probeToken; + const throttleReject = throttleGate.reject; if (throttleReject !== undefined) { + classificationReservation.release(); + classificationReservation = undefined; opts.log("info", `[${opts.label}] [plugin] degenerate-terminal retry answered locally by the 429 hold (#4204) — passing the empty turn through`); return null; } @@ -106,13 +172,55 @@ export function makeContinuationRefetch(opts: ContinuationRetryOpts): () => Prom opts.signal, ); if (!r.response.ok || !r.response.body) { - r.clearTimer(); - if (r.response.status === 429) noteUpstreamThrottle429(opts.upstreamUrl, opts.reqHeaders, r.response.headers.get("retry-after")); - else noteUpstreamThrottleResolved(opts.upstreamUrl, opts.reqHeaders, r.response.status); + if (r.response.status === 429) { + const retryAfter = r.response.headers.get("retry-after"); + const now = Date.now(); + const settle = beginProvisionalUpstreamThrottle429( + opts.upstreamUrl, + opts.reqHeaders, + retryAfter, + now, + requestModel, + probeToken, + classificationReservation, + ); + classificationReservation = undefined; + settleProvisional = settle; + // Keep the fetch watchdog and external-abort listener alive + // until this bounded reader completes. Its only local wait + // is the continuation callback's 250ms latency budget. + const bodyRead = readSmallThrottleErrorBody(r.response) + .then((errorBody) => { + const modelScope = keyringModel429Scope(r.response.status, requestModel, errorBody); + settle(modelScope); + }) + .catch(() => { + settle(); + }) + .finally(() => { + r.clearTimer(); + }) + .catch(() => undefined); + let initialTimer: ReturnType | undefined; + const firstDeadline = new Promise<{ kind: "timeout" }>((resolve) => { + initialTimer = setTimeout(() => resolve({ kind: "timeout" }), MAX_THROTTLE_ERROR_READ_MS); + }); + await Promise.race([ + bodyRead.then(() => ({ kind: "body" as const })), + firstDeadline, + ]).finally(() => clearTimeout(initialTimer)); + } else { + noteUpstreamThrottleResolved(opts.upstreamUrl, opts.reqHeaders, r.response.status, Date.now(), requestModel, probeToken); + classificationReservation.release(); + classificationReservation = undefined; + r.clearTimer(); + } opts.log("warn", `[${opts.label}] [plugin] degenerate-terminal retry rejected (HTTP ${r.response.status}); passing the empty turn through`); return null; } - noteUpstreamThrottleResolved(opts.upstreamUrl, opts.reqHeaders, r.response.status); + noteUpstreamThrottleResolved(opts.upstreamUrl, opts.reqHeaders, r.response.status, Date.now(), requestModel, probeToken); + classificationReservation.release(); + classificationReservation = undefined; const body = r.response.body as ReadableStream; // The retry stream is spliced into a response the caller already // owns, so its own idle timer is dropped — a stalled retry then ends @@ -121,6 +229,9 @@ export function makeContinuationRefetch(opts: ContinuationRetryOpts): () => Prom r.stopIdleTimer(); return body; } catch (e) { + settleProvisional?.(); + classificationReservation?.release(); + releaseUpstreamThrottleProbe(probeToken); opts.log("warn", `[${opts.label}] [plugin] degenerate-terminal retry failed (${e instanceof Error ? e.message : String(e)}); passing the empty turn through`); return null; } diff --git a/src/egress-shed.ts b/src/egress-shed.ts index d3679813fe..3da5a169dd 100644 --- a/src/egress-shed.ts +++ b/src/egress-shed.ts @@ -46,6 +46,11 @@ export interface ShedStats { changed: boolean; } +export interface ShedWireImagesResult { + body: string; + stats: ShedStats; +} + interface ShedOptions { /** Best-effort mNNNNN resolver (session incoming-image index); placeholders quote it. */ resolveRef?: (mediaType: string, b64: string) => Promise; @@ -251,7 +256,7 @@ export async function shedWireImages( protocol: WireProtocol, targetBytes: number, opts: ShedOptions = {}, -): Promise<{ body: string; stats: ShedStats }> { +): Promise { const originalBytes = byteLen(body); const noop: ShedStats = { originalBytes, finalBytes: originalBytes, archived: 0, downscaled: 0, changed: false }; if (targetBytes <= 0 || originalBytes <= targetBytes) return { body, stats: noop }; diff --git a/src/loop/core.ts b/src/loop/core.ts index 93d0da6341..40f1647934 100644 --- a/src/loop/core.ts +++ b/src/loop/core.ts @@ -33,7 +33,7 @@ import { applyConfiguredCompression } from "../external-summary-compress.js"; import { executeSearchContextTarget, resolveDecompress } from "../decompress-shared.js"; import { toolFail, type ProxyToolResult } from "../proxy-tool-result.js"; import { fetchWithRetry, UpstreamHttpError } from "../fetch-util.js"; -import { noteUpstreamThrottle429, noteUpstreamThrottleResolved } from "../upstream-throttle.js"; +import { keyringModel429Scope, noteUpstreamThrottle429, noteUpstreamThrottleResolved } from "../upstream-throttle.js"; import { classifyUpstreamFailure, type SkippedProxyInfo, type UpstreamFailureKind } from "../upstream-fail.js"; import { formatUpstreamError, proxyDispatcher } from "../upstream-proxy.js"; import { warnCacheCollapse } from "../cache-warn.js"; @@ -266,6 +266,8 @@ export interface LoopCtx { export interface RequestOptions { url: string; + /** Exact request model when this wire carries it in a URL/envelope instead of the body. */ + model?: string; headers: Record; /** Final-stage wire transform (e.g. compat.roles role rewrite, #552). * Applied to every outgoing body this loop re-sends — truncation retry, @@ -509,6 +511,7 @@ export async function* runCompressLoop( // #2131: also hand over the exact message count — bodies above the // forensics cap are stored clipped and parse to zero messages. const wireObj = requestOptions.wireTransform ? requestOptions.wireTransform(body) : body; + const requestModel = requestOptions.model ?? (typeof wireObj.model === "string" ? wireObj.model : undefined); // #816 family: a lone surrogate anywhere in the body (model-authored // text, pre-#816 persisted state) serializes as an unpaired \uXXXX // escape and strict upstreams reject the WHOLE body (non-retryable @@ -596,7 +599,10 @@ export async function* runCompressLoop( undefined, signal, (info) => { - if (info.status === 429) noteUpstreamThrottle429(requestOptions.url, requestOptions.headers, null); // retry-after isn't carried in ReplayRetryInfo; the settled response below re-notes with the header + if (info.status === 429) { + const modelScope = keyringModel429Scope(info.status, requestModel, info.detail); + noteUpstreamThrottle429(requestOptions.url, requestOptions.headers, null, Date.now(), modelScope); // Retry-After is not carried on replay metadata. + } // #189: correlate the rejection with the rewrite that // preceded it (shrink ratio + fold point), if any. const lc = lastCompressSuffix(ctx.session.lastCompress); @@ -611,11 +617,12 @@ export async function* runCompressLoop( // (retry-after is not carried on the error) before rethrowing. try { const upstreamSend = await throttledSend; - noteUpstreamThrottleResolved(requestOptions.url, requestOptions.headers, upstreamSend.response.status); + noteUpstreamThrottleResolved(requestOptions.url, requestOptions.headers, upstreamSend.response.status, Date.now(), requestModel); return upstreamSend; } catch (error) { if (error instanceof UpstreamHttpError && error.status === 429) { - noteUpstreamThrottle429(requestOptions.url, requestOptions.headers, null); + const modelScope = keyringModel429Scope(error.status, requestModel, error.body); + noteUpstreamThrottle429(requestOptions.url, requestOptions.headers, null, Date.now(), modelScope); } throw error; } diff --git a/src/server/handle.ts b/src/server/handle.ts index f944ca4cac..26507bc1a2 100644 --- a/src/server/handle.ts +++ b/src/server/handle.ts @@ -471,7 +471,7 @@ async function handleRequest( return; } log("debug", `passthrough: ${req.method ?? "?"} ${maskUrlForLog(req.url ?? "")} — unattributed in-process caller (#1117), relaying verbatim`); - await forward(req, res, opts, scrubCompatDrop(scrubAnthropicPck(protocol, bodyBuffer, log), compatDropPaths, log), null, core, config, log, route, instanceId, undefined); + await forward(req, res, opts, scrubCompatDrop(scrubAnthropicPck(protocol, bodyBuffer, log), compatDropPaths, log), null, core, config, log, route, instanceId, undefined, undefined, undefined, googleModel ?? vertexModel); return; } // #300: bili→bili chain detection. If the inbound request already carries @@ -507,7 +507,7 @@ async function handleRequest( // here would double-manage it. Raw forward, zero state touched. if (headerValue(req, BILI_PLUGIN_BYPASS_HEADER) === "1") { log("debug", `bypass: ${req.method ?? "?"} ${maskUrlForLog(req.url ?? "")} — raw passthrough (legacy in-process compression)`); - await forward(req, res, opts, scrubCompatDrop(scrubAnthropicPck(protocol, bodyBuffer, log), compatDropPaths, log), null, core, config, log, route, instanceId, undefined); + await forward(req, res, opts, scrubCompatDrop(scrubAnthropicPck(protocol, bodyBuffer, log), compatDropPaths, log), null, core, config, log, route, instanceId, undefined, undefined, undefined, googleModel ?? vertexModel); return; } const countTokens = isCountTokensRequest(req.method ?? "GET", urlPath, bodyBuffer.length > 0); @@ -542,7 +542,7 @@ async function handleRequest( v1ConvId = extractConversationId(unwrapped.inner) ?? undefined; if (isV1BackgroundSideRequest(unwrapped.meta, unwrapped.inner)) { log("info", `v1internal background utility call (requestType=${typeof unwrapped.meta.requestType === "string" ? unwrapped.meta.requestType : "micro"}) — verbatim passthrough, ACP state machine bypassed (#3129)`); - await forward(req, res, opts, scrubCompatDrop(scrubAnthropicPck(protocol, bodyBuffer, log), compatDropPaths, log), null, core, config, log, route, instanceId, undefined); + await forward(req, res, opts, scrubCompatDrop(scrubAnthropicPck(protocol, bodyBuffer, log), compatDropPaths, log), null, core, config, log, route, instanceId, undefined, undefined, undefined, v1Model ?? googleModel ?? vertexModel); return; } log("debug", `v1internal envelope unwrapped (model=${v1Model ?? "?"} convId=${v1ConvId ?? "none"})`); @@ -618,6 +618,12 @@ async function handleRequest( // relay verbatim, exactly like the unparseable-body path — the upstream // rejects its own API contract. (#806's crash cannot recur: a verbatim // forward never enters the kernel.) + // Resolve both the context model and exact wire model before any verbatim + // path can forward without a Prepared request. + const bodyModel = parsed && typeof parsed === "object" && "model" in parsed && typeof parsed.model === "string" ? parsed.model : undefined; + const vertexPathModel = vertexModel !== undefined ? normalizeVertexModelId(vertexModel) : undefined; + const requestModel = bodyModel ?? googleModel ?? vertexPathModel ?? v1Model; + const throttleModel = v1Model ?? googleModel ?? vertexModel ?? bodyModel; if ((protocol === "anthropic" || protocol === "openai") && parsed !== null) { const p = typeof parsed === "object" && !Array.isArray(parsed) ? parsed as Record : undefined; if (!p || !Array.isArray(p.messages)) { @@ -629,28 +635,21 @@ async function handleRequest( } log("warn", `[${protocol}] body has no "messages" array — not a model conversation; relaying verbatim to ${maskUrlsInText(upstreamOrigin)} instead of rejecting (#1284) — ${req.method ?? "?"} ${maskUrlForLog(req.url ?? "")}`); } - await forward(req, res, opts, scrubCompatDrop(scrubAnthropicPck(protocol, bodyBuffer, log), compatDropPaths, log), null, core, config, log, route, instanceId, undefined); + await forward(req, res, opts, scrubCompatDrop(scrubAnthropicPck(protocol, bodyBuffer, log), compatDropPaths, log), null, core, config, log, route, instanceId, undefined, undefined, undefined, throttleModel); return; } } - // The effective model for this request. Every wire carries it in the body - // except Gemini, whose URL path holds it (`/v1beta/models/:…`), and - // Anthropic-on-Vertex, whose path holds it the same way - // (`…/publishers/anthropic/models/:rawPredict`) with no body `model` - // at all. - const bodyModel = parsed && typeof parsed === "object" && "model" in parsed && typeof parsed.model === "string" ? parsed.model : undefined; + // Only the path-derived Vertex id is normalised: it is the one id whose // spelling the wire itself guarantees (the publisher path), and dropping an // `@version` tag is what makes the registry lookup and the compress // thresholds land on the right row. - const vertexPathModel = vertexModel !== undefined ? normalizeVertexModelId(vertexModel) : undefined; + if (vertexModel !== undefined && vertexPathModel !== undefined && vertexPathModel !== vertexModel && !vertexModelNormalised.has(vertexModel)) { if (vertexModelNormalised.size < VERTEX_MODEL_LOG_CAP) vertexModelNormalised.add(vertexModel); log("info", `[anthropic] Vertex model id normalised ${vertexModel} -> ${vertexPathModel}; the @version tag is not part of the publisher model id on this wire`); } - // #3129: v1internal carries the model in the OUTER envelope, not the path - // (/v1internal:* has no /models/: segment) and not always the inner body. - const requestModel = bodyModel ?? googleModel ?? vertexPathModel ?? v1Model; + // Capture the CLIENT's raw incoming request (before bili rebuilds) to // resolve whether codex sends previous_response_id + full input vs delta. if (opts.debug && parsed && typeof parsed === "object") { @@ -2240,13 +2239,21 @@ async function handleRequest( } } sideBody = scrubCompatDrop(sideBody, compatDropPaths, log); + // Gemini and Vertex encode stream mode in the path rather than the body. + const requestStream = parsed !== null && typeof parsed === "object" && !Array.isArray(parsed) + && "stream" in parsed && parsed.stream === true; const sidePrepared: Prepared = { body: sideBody, session, processedMessages: [], originalMessages: [], protocol, - stream: (parsed as { stream?: unknown }).stream === true, + model: throttleModel, + stream: protocol === "google" + ? googlePathKind(urlPath) === "stream-generate" + : protocol === "anthropic" && vertexModel !== undefined + ? urlPath.includes(":streamRawPredict") + : requestStream, compressInjected: false, sidePassthrough: true, }; @@ -2886,7 +2893,7 @@ async function handleRequest( clearSignedRefusal(guardScheme); log("warn", `[signed-passthrough] built-in-scheme request without a working re-sign arm forwarded byte-untouched, no compression (#1884 escape hatch via resign["sdk-hmac-sha256"].passthrough / BILI_RESIGN_PASSTHROUGH — the user has acknowledged this link runs uncompressed)`); forwarded = true; - await forward(req, res, opts, bodyBuffer, null, core, reqConfig, log, route, instanceId, undefined); + await forward(req, res, opts, bodyBuffer, null, core, reqConfig, log, route, instanceId, undefined, undefined, undefined, throttleModel); return; } recordSignedRefusal(guardScheme, upstreamOrigin); @@ -2922,7 +2929,7 @@ async function handleRequest( return next; } : undefined; - await forward(req, res, opts, pendingForward.body, pendingForward.prepared, core, reqConfig, log, route, instanceId, affinity, overflowRefold, v1Meta); + await forward(req, res, opts, pendingForward.body, pendingForward.prepared, core, reqConfig, log, route, instanceId, affinity, overflowRefold, v1Meta, throttleModel); // Remember for ALL modes (not just plugin): wire clients (dsh, // hermes, unplug'd pi) read the same panel via /__bili/plugin/status // and need the nudge/breakdown sections too; locked so a racing @@ -2946,6 +2953,6 @@ async function handleRequest( if (protocol === null && !opts.passthrough && !routePassthrough && !isModelDiscoveryPath(urlPath) && !isV1InternalControlPlanePath(urlPath)) { logUnrecognizedPath(log, req.url ?? ""); } - await forward(req, res, opts, scrubCompatDrop(scrubAnthropicPck(protocol, bodyBuffer, log), compatDropPaths, log), null, core, reqConfig, log, route, instanceId, undefined); + await forward(req, res, opts, scrubCompatDrop(scrubAnthropicPck(protocol, bodyBuffer, log), compatDropPaths, log), null, core, reqConfig, log, route, instanceId, undefined, undefined, undefined, throttleModel); } } diff --git a/src/server/relay.ts b/src/server/relay.ts index fa64fc25b3..0711cb8642 100644 --- a/src/server/relay.ts +++ b/src/server/relay.ts @@ -25,7 +25,7 @@ import { detectGenerationBoundary, GENERATION_BOUNDARY_NOTE } from "../generatio import { findRoute, resolveDeclaredProtocol, type ProxyOptions } from "../config.js"; import { makeContinuationRefetch } from "../degenerate-retry.js"; import { dumpRejectedBody } from "../error-dump.js"; -import { extractWsOversizeError, formatMiB, shedEgressIfOverBudget, shedWireImages, shedWireImagesToCount, type ShedStats } from "../egress-shed.js"; +import { extractWsOversizeError, formatMiB, shedEgressIfOverBudget, shedWireImages, shedWireImagesToCount, type ShedStats, type ShedWireImagesResult } from "../egress-shed.js"; import { conversationPromptSections, externalSummaryEnabled, messageTimestampSections } from "../external-summary-surface.js"; import { fakeBufCap, injectFakeCompletionHint, isFakeCompletion, maxFakeCompletionRetries } from "../fake-completion.js"; import { fetchWithTimeout, fetchWithTransportRetry } from "../fetch-util.js"; @@ -58,7 +58,7 @@ import { safePrefix, safeSuffix, scrubLoneSurrogatesOnWire } from "../text-safe. import { clearUpstreamAlertsForHost, noteCircuitFailFastLog, recordUpstreamAlert, recordUpstreamCircuitFailure, recordUpstreamCircuitSuccess, upstreamCircuitBlockReason, upstreamCircuitLastFailureKind, upstreamCircuitRetryAfterSeconds } from "../upstream-alerts.js"; import { describeUpstreamFailure, formatUpstreamError, proxyDispatcher, recordUpstreamConnection } from "../upstream-proxy.js"; import { markExpectedFailFast, withUpstreamFailure, type UpstreamFailureKind } from "../upstream-fail.js"; -import { noteUpstreamThrottle429, noteUpstreamThrottleResolved, upstreamThrottleLocalReject } from "../upstream-throttle.js"; +import { keyringModel429Scope, noteUpstreamThrottle429, noteUpstreamThrottleResolved, releaseUpstreamThrottleProbe, requestModelFromBody, upstreamThrottleGate } from "../upstream-throttle.js"; import { applyEstimateCalibration, currentCalibrationFactor, inspectContextOverflow, isMarkerlessOverflowReject, MIN_ASSUMED_WINDOW_TOKENS, normalizeUpstreamOrigin, usageOutputTotal, usageTotals, type ContextOverflowInfo, type WireProtocol } from "../util.js"; import { billOutputUsage } from "../billing.js"; import { DUMP_PLAINTEXT_BANNER } from "../dump-banner.js"; @@ -269,6 +269,7 @@ async function forwardCaptured( affinity?: string, overflowRefold?: (realWindow: number | undefined) => Promise<{ body: string | Buffer; prepared: Prepared | null } | null>, v1Meta?: Record, + throttleModelHint?: string, ): Promise { const forwardStartedAt = Date.now(); const reply = prepared && !prepared.sidePassthrough ? createHandoffReply(prepared.session, () => !res.destroyed) : undefined; @@ -846,14 +847,15 @@ async function forwardCaptured( hint: "upstream circuit open — the endpoint is unreachable and bili is failing fast instead of resending full context until a probe succeeds", })); } - // #4204: egress-side 429 throttle governor — while consecutive sends to this - // origin+credential came back 429, answer LOCALLY (429 + Retry-After under - // budget, then a terminal 424 outside every known SDK retry set) instead of - // forwarding: a host-driven retry loop must not be amplified into egress - // that pins a concurrency-style provider limit for the whole account. Zero - // egress while held; one probe per backoff window keeps recovery automatic. + // #4204: account-wide holds stay origin+credential scoped. KeyRing's + // request-matched model-temporarily-unavailable error gets a separate + // origin+credential+model hold so unrelated models remain eligible. + const throttleModel = throttleModelHint ?? (typeof prepared?.model === "string" ? prepared.model : requestModelFromBody(wireBody)); + let throttleProbeToken: symbol | undefined; if (req.method !== "GET" && req.method !== "HEAD") { - const throttleReject = upstreamThrottleLocalReject(upstreamUrl, headers); + const throttleGate = upstreamThrottleGate(upstreamUrl, headers, Date.now(), throttleModel); + throttleProbeToken = throttleGate.probeToken; + const throttleReject = throttleGate.reject; if (throttleReject !== undefined) { log(throttleReject.status === 424 ? "warn" : "debug", `[${prepared?.session.id ?? "passthrough"}] [acp-proxy] 429-hold: answered locally with ${throttleReject.status} without egress (#4204)`); if (res.headersSent) emitStreamError(res, prepared?.protocol ?? "openai", throttleReject.message, undefined, opts.streamErrorShape); @@ -861,7 +863,7 @@ async function forwardCaptured( res.writeHead(throttleReject.status, throttleReject.retryAfterSec !== null ? { "content-type": "application/json", "retry-after": String(throttleReject.retryAfterSec) } : { "content-type": "application/json" }); - res.end(JSON.stringify({ error: { type: "rate_limit_error", code: throttleReject.code, stage: "forward", message: throttleReject.message, retryable: throttleReject.status === 429 } })); + res.end(JSON.stringify({ error: { type: "rate_limit_error", code: throttleReject.code, stage: "forward", message: throttleReject.message, retryable: throttleReject.status === 429, ...(throttleReject.model === undefined ? {} : { scope: "model", model: throttleReject.model }) } })); } return; } @@ -892,6 +894,7 @@ async function forwardCaptured( } catch (error) { const oversize = extractWsOversizeError(error); if (oversize === undefined) { + releaseUpstreamThrottleProbe(throttleProbeToken); recordUpstreamConnection(upstreamUrl, proxyUrl, error, skippedProxy); recordUpstreamAlert(upstreamUrl, error, proxyUrl !== undefined, skippedProxy); const failure = describeUpstreamFailure(error, upstreamUrl, proxyUrl, skippedProxy); @@ -918,11 +921,22 @@ async function forwardCaptured( const budget = limitsWsOutboundBytes(); const target = Math.min(sentBytes * 0.85, budget > 0 ? budget : Number.MAX_SAFE_INTEGER); if (prepared === null || prepared === undefined || prepared.sidePassthrough || prepared.protocol === undefined || typeof wireBody !== "string" || sentBytes <= 0 || !res.writable || res.writableEnded || res.destroyed) { + releaseUpstreamThrottleProbe(throttleProbeToken); throw new Error(wsOversizeDiagnosis(sentBytes, undefined, budget)); } const shedSession = prepared.session; - const { body: shrunk, stats } = await shedWireImages(wireBody, prepared.protocol, target, { resolveRef: (mt, b64) => refForItemBytes(shedSession, mt, b64) }); - if (!stats.changed || stats.finalBytes >= sentBytes) throw new Error(wsOversizeDiagnosis(sentBytes, stats, budget)); + let shedResult: ShedWireImagesResult; + try { + shedResult = await shedWireImages(wireBody, prepared.protocol, target, { resolveRef: (mt, b64) => refForItemBytes(shedSession, mt, b64) }); + } catch (error) { + releaseUpstreamThrottleProbe(throttleProbeToken); + throw error; + } + const { body: shrunk, stats } = shedResult; + if (!stats.changed || stats.finalBytes >= sentBytes) { + releaseUpstreamThrottleProbe(throttleProbeToken); + throw new Error(wsOversizeDiagnosis(sentBytes, stats, budget)); + } log("info", `[${prepared.session.id}] [ws-byte-shed] upstream closed 1009 with zero events at ${formatMiB(sentBytes)} MiB — shed to ${formatMiB(stats.finalBytes)} MiB (archived=${stats.archived} downscaled=${stats.downscaled}), resending once within this request`); try { const shrunkWire = scrubLoneSurrogatesOnWire(shrunk); @@ -959,6 +973,7 @@ async function forwardCaptured( } } catch (retryErr) { log("warn", `[${prepared?.session.id ?? "unknown"}] [ws-byte-shed] shrunken resend failed: ${formatUpstreamError(retryErr, upstreamUrl, proxyUrl, skippedProxy)}`); + releaseUpstreamThrottleProbe(throttleProbeToken); throw new Error(wsOversizeDiagnosis(sentBytes, stats, budget), { cause: retryErr }); } } @@ -1616,14 +1631,16 @@ async function forwardCaptured( if (bodyText.length > 600) snippet += " …"; if (!snippet) snippet = "(no body)"; loggerLog("warn", `[${errSid}] ← upstream ${upstream.status}${reqIdText}: ${snippet}`); - // #4204: feed the egress throttle governor with this send's FINAL verdict - // (one observation per request — in-request refolds/resends above all - // converge into this same response handling). A 429 arms/extends the - // hold; ANY other resolved status proves the endpoint is processing - // again and clears it. + // #4204: feed this send's FINAL verdict. A model-scoped KeyRing 429 is + // selected only by its structured code plus exact request-model match; + // keyring_all_limited and every ambiguous response remain account-wide. if (req.method !== "GET" && req.method !== "HEAD") { - if (upstream.status === 429) noteUpstreamThrottle429(upstreamUrl, headers, upstream.headers.get("retry-after")); - else noteUpstreamThrottleResolved(upstreamUrl, headers, upstream.status); + if (upstream.status === 429) { + const modelScope = keyringModel429Scope(upstream.status, throttleModel, errBody); + noteUpstreamThrottle429(upstreamUrl, headers, upstream.headers.get("retry-after"), Date.now(), modelScope, throttleProbeToken); + } else { + noteUpstreamThrottleResolved(upstreamUrl, headers, upstream.status, Date.now(), throttleModel, throttleProbeToken); + } } // #762: persist the exact forwarded body on 4xx (env-gated: BILI_DUMP_4XX=1). if (upstream.status >= 400 && upstream.status < 500) { @@ -1668,11 +1685,9 @@ async function forwardCaptured( if (prepared !== null && prepared.responseExtraHeaders !== undefined) Object.assign(respHeaders, prepared.responseExtraHeaders); res.writeHead(upstream.status, respHeaders); } - // #4204: success-side counterpart of the non-2xx observation above — a - // resolved 2xx proves the endpoint is accepting sends again, so clear any - // hold on this origin+credential (counterpart needed because the non-ok - // branch returns early and never reaches this point). - if (req.method !== "GET" && req.method !== "HEAD") noteUpstreamThrottleResolved(upstreamUrl, headers, upstream.status); + // #4204: a resolved response preserves the existing account-hold recovery + // signal and clears only the matching model hold. + if (req.method !== "GET" && req.method !== "HEAD") noteUpstreamThrottleResolved(upstreamUrl, headers, upstream.status, Date.now(), throttleModel, throttleProbeToken); // #3376: idle prompt-cache keepalive — snapshot the EXACT accepted bytes // and arm the TTL-refresh timer. Pipeline-prepared requests only // (passthrough stays out of scope); internally gated to anthropic-wire @@ -1804,6 +1819,7 @@ async function forwardCaptured( makeContinuationRefetch({ protocol: "responses", body, + model: throttleModel, upstreamUrl, reqHeaders: buildForwardHeaders(headers), resign: applyResign, proxyUrl, @@ -1832,6 +1848,7 @@ async function forwardCaptured( makeContinuationRefetch({ protocol: prepared.protocol, body, + model: throttleModel, upstreamUrl, reqHeaders: buildForwardHeaders(headers), resign: applyResign, proxyUrl, @@ -1965,6 +1982,7 @@ async function forwardCaptured( makeContinuationRefetch({ protocol: "responses", body, + model: throttleModel, upstreamUrl, reqHeaders: buildForwardHeaders(headers), resign: applyResign, proxyUrl, @@ -2010,6 +2028,7 @@ async function forwardCaptured( makeContinuationRefetch({ protocol: "responses", body, + model: throttleModel, upstreamUrl, reqHeaders: buildForwardHeaders(headers), resign: applyResign, proxyUrl, @@ -2034,6 +2053,7 @@ async function forwardCaptured( makeContinuationRefetch({ protocol: p.protocol, body, + model: throttleModel, wrapBody: v1Active ? v1Wrap : undefined, upstreamUrl, reqHeaders: buildForwardHeaders(headers), resign: applyResign, @@ -2217,7 +2237,7 @@ async function forwardCaptured( // #3280: egress goes out enveloped via wireTransform, so every // round-2+ response comes back enveloped too — unwrap at the loop's // single adoption point or parseStream sees no candidates. - { url: upstreamUrl, headers: reqHeaders, wireTransform, resign: applyResign, responseStreamTransform: v1Active ? v1UnwrapStream : undefined }, + { url: upstreamUrl, headers: reqHeaders, model: throttleModel, wireTransform, resign: applyResign, responseStreamTransform: v1Active ? v1UnwrapStream : undefined }, adapter, systemPrompt, clientAbort.signal, diff --git a/src/upstream-throttle.ts b/src/upstream-throttle.ts index 50f07265a8..1afcff5542 100644 --- a/src/upstream-throttle.ts +++ b/src/upstream-throttle.ts @@ -7,8 +7,10 @@ // and pinned the WHOLE account at the limit for minutes — every other session // failed too, until the client (and proxy process) was restarted. // -// Fix: when consecutive sends to one origin (+ credential) come back 429, stop -// amplifying. Further sends are answered LOCALLY instead of being forwarded: +// Fix: consecutive 429s create a local hold by origin+credential. A verified +// KeyRing model-endpoint refusal is keyed by origin+credential+request-model; +// account quota and every ambiguous/unrecognized 429 remain origin+credential. +// Further sends covered by that scope are answered LOCALLY instead of forwarded: // - while under the budget: 429 + a Retry-After header (exponential backoff // ×2 per consecutive 429, jittered, capped; an upstream-provided // Retry-After wins when larger) — zero egress while held, so the provider @@ -16,9 +18,9 @@ // - once the budget is exhausted: a TERMINAL 424 — deliberately outside // every known client-SDK retry set ({408, 409, 429, ≥500}), so hosts give // up immediately and surface the error for a manual resend. -// One request per backoff window is admitted as a probe, so recovery is -// AUTOMATIC without a restart; ANY resolved non-429 response (including 400/401) -// proves the endpoint is processing again and clears the hold at once. +// One request per backoff window is admitted as a probe. A completion can +// release only slots owned by its admission token; resolved non-429s clear +// confirmed account holds, while a model hold clears only for that same model. // // Deliberately separate from the #3148 circuit breaker: that one fails fast on // PRE-RESPONSE transport deaths (endpoint unreachable); this one governs a @@ -44,19 +46,253 @@ const PROBE_INFLIGHT_TTL_MS = 30_000; /** An upstream Retry-After beyond this is treated as "later than we will * locally wait" — the hold still arms at our own backoff ceiling. */ const RETRY_AFTER_CAP_MS = 600_000; +/** Only parse small structured errors; large/malformed bodies keep legacy scope. */ +const MAX_KEYRING_ERROR_BODY_BYTES = 16 * 1024; +const KEYRING_MODEL_LIMIT_CODE = "keyring_model_temporarily_unavailable"; + +function recordOf(value: unknown): Record | undefined { + return value !== null && typeof value === "object" && !Array.isArray(value) + ? value as Record + : undefined; +} + +function parseSmallJsonRecord(body: string | Uint8Array): Record | undefined { + if (body.length > MAX_KEYRING_ERROR_BODY_BYTES) return undefined; + try { + const text = typeof body === "string" ? body : new TextDecoder().decode(body); + return recordOf(JSON.parse(text) as unknown); + } catch { + return undefined; + } +} + +function jsonCodeAt(body: string | Uint8Array, index: number): number { + if (typeof body === "string") return index < body.length ? body.charCodeAt(index) : -1; + return index < body.length ? body[index] ?? -1 : -1; +} + +function skipJsonWhitespace(body: string | Uint8Array, start: number): number { + let index = start; + while (index < body.length) { + const code = jsonCodeAt(body, index); + if (code !== 0x20 && code !== 0x09 && code !== 0x0a && code !== 0x0d) break; + index += 1; + } + return index; +} + +function skipJsonString(body: string | Uint8Array, start: number): number | undefined { + if (jsonCodeAt(body, start) !== 0x22) return undefined; + let index = start + 1; + while (index < body.length) { + const code = jsonCodeAt(body, index); + if (code === 0x22) return index + 1; + if (code < 0x20) return undefined; + if (code === 0x5c) { + index += 1; + if (index >= body.length) return undefined; + const escaped = jsonCodeAt(body, index); + if (escaped === 0x75) { + for (let digit = index + 1; digit <= index + 4; digit += 1) { + const value = jsonCodeAt(body, digit); + if (!((value >= 0x30 && value <= 0x39) || (value >= 0x41 && value <= 0x46) || (value >= 0x61 && value <= 0x66))) return undefined; + } + index += 5; + continue; + } + if (escaped !== 0x22 && escaped !== 0x5c && escaped !== 0x2f + && escaped !== 0x62 && escaped !== 0x66 && escaped !== 0x6e && escaped !== 0x72 && escaped !== 0x74) return undefined; + } + index += 1; + } + return undefined; +} + +function readJsonString(body: string | Uint8Array, start: number, end: number): string | undefined { + try { + const source = typeof body === "string" ? body.slice(start, end) : new TextDecoder().decode(body.subarray(start, end)); + const value = JSON.parse(source) as unknown; + return typeof value === "string" ? value : undefined; + } catch { + return undefined; + } +} + +function skipJsonNumber(body: string | Uint8Array, start: number): number | undefined { + let index = start; + if (jsonCodeAt(body, index) === 0x2d) index += 1; + const integer = jsonCodeAt(body, index); + if (integer === 0x30) { + index += 1; + } else if (integer >= 0x31 && integer <= 0x39) { + index += 1; + while (jsonCodeAt(body, index) >= 0x30 && jsonCodeAt(body, index) <= 0x39) index += 1; + } else { + return undefined; + } + if (jsonCodeAt(body, index) === 0x2e) { + index += 1; + const fraction = index; + while (jsonCodeAt(body, index) >= 0x30 && jsonCodeAt(body, index) <= 0x39) index += 1; + if (index === fraction) return undefined; + } + const exponent = jsonCodeAt(body, index); + if (exponent === 0x65 || exponent === 0x45) { + index += 1; + const sign = jsonCodeAt(body, index); + if (sign === 0x2b || sign === 0x2d) index += 1; + const digits = index; + while (jsonCodeAt(body, index) >= 0x30 && jsonCodeAt(body, index) <= 0x39) index += 1; + if (index === digits) return undefined; + } + return index; +} + +function skipJsonValue(body: string | Uint8Array, start: number, depth = 0): number | undefined { + if (depth > 256) return undefined; + const index = skipJsonWhitespace(body, start); + const code = jsonCodeAt(body, index); + if (code === 0x22) return skipJsonString(body, index); + if (code === 0x5b || code === 0x7b) { + const object = code === 0x7b; + const close = object ? 0x7d : 0x5d; + let cursor = skipJsonWhitespace(body, index + 1); + if (jsonCodeAt(body, cursor) === close) return cursor + 1; + while (cursor < body.length) { + if (object) { + const keyEnd = skipJsonString(body, cursor); + if (keyEnd === undefined) return undefined; + cursor = skipJsonWhitespace(body, keyEnd); + if (jsonCodeAt(body, cursor) !== 0x3a) return undefined; + cursor += 1; + } + const valueEnd = skipJsonValue(body, cursor, depth + 1); + if (valueEnd === undefined) return undefined; + cursor = skipJsonWhitespace(body, valueEnd); + const separator = jsonCodeAt(body, cursor); + if (separator === close) return cursor + 1; + if (separator !== 0x2c) return undefined; + cursor = skipJsonWhitespace(body, cursor + 1); + } + return undefined; + } + if (code === 0x74 && jsonCodeAt(body, index + 1) === 0x72 && jsonCodeAt(body, index + 2) === 0x75 && jsonCodeAt(body, index + 3) === 0x65) return index + 4; + if (code === 0x66 && jsonCodeAt(body, index + 1) === 0x61 && jsonCodeAt(body, index + 2) === 0x6c && jsonCodeAt(body, index + 3) === 0x73 && jsonCodeAt(body, index + 4) === 0x65) return index + 5; + if (code === 0x6e && jsonCodeAt(body, index + 1) === 0x75 && jsonCodeAt(body, index + 2) === 0x6c && jsonCodeAt(body, index + 3) === 0x6c) return index + 4; + return skipJsonNumber(body, index); +} + +/** Extract only the top-level model without materializing large passthrough bodies. */ +function scanRequestModel(body: string | Uint8Array): string | undefined { + let index = skipJsonWhitespace(body, 0); + if (jsonCodeAt(body, index) !== 0x7b) return undefined; + index = skipJsonWhitespace(body, index + 1); + let model: string | undefined; + if (jsonCodeAt(body, index) === 0x7d) return undefined; + while (index < body.length) { + const keyStart = index; + const keyEnd = skipJsonString(body, keyStart); + if (keyEnd === undefined) return undefined; + const key = keyEnd - keyStart <= 32 ? readJsonString(body, keyStart, keyEnd) : undefined; + index = skipJsonWhitespace(body, keyEnd); + if (jsonCodeAt(body, index) !== 0x3a) return undefined; + const valueStart = skipJsonWhitespace(body, index + 1); + let valueEnd: number | undefined; + if (key === "model" && jsonCodeAt(body, valueStart) === 0x22) { + valueEnd = skipJsonString(body, valueStart); + if (valueEnd === undefined) return undefined; + model = readJsonString(body, valueStart, valueEnd); + } else { + if (key === "model") model = undefined; + valueEnd = skipJsonValue(body, valueStart); + if (valueEnd === undefined) return undefined; + } + index = skipJsonWhitespace(body, valueEnd); + const separator = jsonCodeAt(body, index); + if (separator === 0x7d) return skipJsonWhitespace(body, index + 1) === body.length ? model : undefined; + if (separator !== 0x2c) return undefined; + index = skipJsonWhitespace(body, index + 1); + } + return undefined; +} + +/** Read the top-level request model without materializing a large passthrough body. */ +export function requestModelFromBody(body: string | Uint8Array): string | undefined { + const model = scanRequestModel(body); + return typeof model === "string" && model.length > 0 ? model : undefined; +} + +/** + * KeyRing emits this code only for a model-endpoint refusal after fresh usage + * confirms account headroom. Bili's local relay hold also carries an explicit + * model scope. Recognize either only when the structured marker and model both + * match the exact request; malformed bodies and every mismatch remain key-wide. + */ +export function keyringModel429Scope( + status: number, + requestModel: string | undefined, + responseBody: string | Uint8Array | null | undefined, +): string | undefined { + if (status !== 429 || !requestModel || responseBody == null) return undefined; + const error = recordOf(parseSmallJsonRecord(responseBody)?.error); + if (error?.model !== requestModel) return undefined; + if (error.code === KEYRING_MODEL_LIMIT_CODE) return requestModel; + if (error.code === "bili_throttle_hold" && error.scope === "model") return requestModel; + return undefined; +} + +function modelHoldKey(key: string, model: string): string { + return `${key}\u0000model\u0000${model}`; +} + +type HoldSlotKey = string | symbol; interface HoldState { streak: number; nextAttemptAt: number; - /** When the last admitted probe was released (0 = none outstanding). */ - probeAt: number; + /** Admission token and time for the probe currently owning this slot. */ + probeToken: symbol | undefined; + probeAt: number | undefined; host: string; cred: string | undefined; + /** Account key shared by the account, model, and provisional group. */ + accountKey: string; + /** Present only after the refusal is verified as model-scoped. */ + model?: string; + /** A pending body refinement remains account-scoped until it settles. */ + provisional: boolean; + provisionalRequestModel?: string; } -const holds = new Map(); +export interface ThrottleClassificationReservation { + release(): void; +} + +interface ClassificationReservationState { + accountKey: string; + active: boolean; + claimed: boolean; + onRelease?: () => void; +} + +interface ProvisionalGroup { + /** One account-scoped gate shared by every pending observation for a key. */ + state: HoldState; + /** Individual records keep each observation's settlement independent. */ + observations: Set; +} + +const holds = new Map(); +const provisionalGroups = new Map(); +const MAX_CLASSIFICATIONS_PER_ACCOUNT = 64; +const MAX_CLASSIFICATION_ACCOUNTS = 64; +const activeClassificationCounts = new Map(); +const classificationReservations = new Map(); +const NOOP_CLASSIFICATION_RESERVATION: ThrottleClassificationReservation = { + release(): void {}, +}; -interface ThrottleLocalReject { +export interface ThrottleLocalReject { /** 429 while under budget (Retry-After attached); 424 terminal. */ status: 429 | 424; code: "bili_throttle_hold" | "bili_throttle_terminal"; @@ -66,6 +302,13 @@ interface ThrottleLocalReject { maxConsecutive: number; nextAttemptAt: number; host: string; + model?: string; +} + +export interface UpstreamThrottleGate { + reject?: ThrottleLocalReject; + /** Present only when this send claimed one or more expired hold slots. */ + probeToken?: symbol; } function parseRetryAfterMs(headerValue: string | null, now: number): number | null { @@ -91,130 +334,532 @@ function holdKey(url: string, headers: Record): { key: string; h return { key: cred === undefined ? host : `${host}\u0000${cred}`, host, cred }; } -function evictOldest(): void { - const oldest = holds.keys().next().value; - if (oldest !== undefined) holds.delete(oldest); +function releaseClassificationReservation( + reservation: ThrottleClassificationReservation, + state: ClassificationReservationState, +): void { + if (!state.active) return; + state.active = false; + classificationReservations.delete(reservation); + + const activeCount = activeClassificationCounts.get(state.accountKey); + if (activeCount === undefined || activeCount <= 1) { + activeClassificationCounts.delete(state.accountKey); + } else { + activeClassificationCounts.set(state.accountKey, activeCount - 1); + } + + const onRelease = state.onRelease; + state.onRelease = undefined; + onRelease?.(); +} + +/** + * Reserve one classification slot before forwarding a continuation. + * `undefined` means capacity is saturated; disabled throttling returns a no-op reservation. + */ +export function reserveUpstreamThrottleClassification( + url: string, + headers: Record, +): ThrottleClassificationReservation | undefined { + if (!throttle429Enabled()) return NOOP_CLASSIFICATION_RESERVATION; + const { key } = holdKey(url, headers); + const activeCount = activeClassificationCounts.get(key) ?? 0; + if ( + activeCount >= MAX_CLASSIFICATIONS_PER_ACCOUNT + || (activeCount === 0 && activeClassificationCounts.size >= MAX_CLASSIFICATION_ACCOUNTS) + ) return undefined; + + activeClassificationCounts.set(key, activeCount + 1); + const state: ClassificationReservationState = { + accountKey: key, + active: true, + claimed: false, + }; + let reservation: ThrottleClassificationReservation; + reservation = { + release(): void { + releaseClassificationReservation(reservation, state); + }, + }; + classificationReservations.set(reservation, state); + return reservation; +} + +function evictOldestModel(): boolean { + for (const [slotKey, state] of holds) { + if (state.model === undefined || state.provisional) continue; + holds.delete(slotKey); + return true; + } + return false; +} + +function evictOldestAccountWide(): boolean { + for (const [slotKey, state] of holds) { + if (state.model !== undefined || state.provisional) continue; + holds.delete(slotKey); + return true; + } + return false; +} + +/** + * Keep the 64-slot table bounded. A new account verdict evicts the oldest + * model slot first, then the oldest confirmed account slot if necessary. + * Model-local and provisional evidence can never displace confirmed accounts. + */ +function storeHold(slotKey: HoldSlotKey, state: HoldState, preserveAccountWide: boolean): boolean { + if (!holds.has(slotKey) && holds.size >= MAX_HOLDS) { + if (state.model !== undefined || state.provisional || preserveAccountWide) { + if (!evictOldestModel()) return false; + } else if (!evictOldestModel() && !evictOldestAccountWide()) { + return false; + } + } + holds.set(slotKey, state); + return true; } function makeReject(st: HoldState, now: number): ThrottleLocalReject { const maxC = throttle429MaxConsecutive(); const terminal = st.streak >= maxC; const secsLeft = Math.max(1, Math.ceil((st.nextAttemptAt - now) / 1000)); + const modelText = st.model === undefined ? "" : ` for model "${st.model}"`; if (terminal) { const capSec = Math.ceil(throttle429MaxDelayMs() / 1000); return { status: 424, code: "bili_throttle_terminal", - message: `upstream rate-limited ${st.streak} consecutive request(s) to ${st.host}; bili stops amplifying after ${maxC} consecutive rate limits — resend manually once the provider recovers (a background probe keeps checking at most every ${capSec}s) (#4204)`, + message: `upstream rate-limited ${st.streak} consecutive request(s) to ${st.host}${modelText}; bili stops amplifying after ${maxC} consecutive rate limits — resend manually once the provider recovers (a background probe keeps checking at most every ${capSec}s) (#4204)`, retryAfterSec: null, streak: st.streak, maxConsecutive: maxC, nextAttemptAt: st.nextAttemptAt, host: st.host, + ...(st.model === undefined ? {} : { model: st.model }), }; } return { status: 429, code: "bili_throttle_hold", - message: `upstream rate-limited ${st.streak} consecutive request(s) to ${st.host}; bili is holding outbound sends until ${new Date(st.nextAttemptAt).toISOString()} (~${secsLeft}s, exponential backoff) instead of amplifying — zero egress while held, auto-resumes when the limit clears (#4204)`, + message: `upstream rate-limited ${st.streak} consecutive request(s) to ${st.host}${modelText}; bili is holding outbound sends until ${new Date(st.nextAttemptAt).toISOString()} (~${secsLeft}s, exponential backoff) instead of amplifying — zero egress while held, auto-resumes when the limit clears (#4204)`, retryAfterSec: secsLeft, streak: st.streak, maxConsecutive: maxC, nextAttemptAt: st.nextAttemptAt, host: st.host, + ...(st.model === undefined ? {} : { model: st.model }), + }; +} + +function createHoldState( + accountKey: string, + host: string, + cred: string | undefined, + retryAfterHeader: string | null, + now: number, + previous: HoldState | undefined, + model: string | undefined, + provisional: boolean, + provisionalRequestModel?: string, +): HoldState { + const streak = (previous?.streak ?? 0) + 1; + const base = throttle429BaseDelayMs(); + const cap = throttle429MaxDelayMs(); + const exp = Math.min(cap, base * 2 ** Math.min(streak - 1, 30)); + const jittered = exp * (0.5 + Math.random() * 0.5); + const raMs = parseRetryAfterMs(retryAfterHeader, now); + const delay = Math.max(jittered, raMs ?? 0); + return { + streak, + nextAttemptAt: now + delay, + probeAt: previous?.probeAt, + probeToken: previous?.probeToken, + host, + cred, + accountKey, + ...(model === undefined ? {} : { model }), + provisional, + ...(provisionalRequestModel === undefined ? {} : { provisionalRequestModel }), }; } -/** Pre-forward gate. Returns undefined = allow the send (and, when a backoff - * window just opened, marks the single in-flight probe slot). Returns a local - * reject descriptor when the send must be answered WITHOUT contacting the - * upstream. */ -export function upstreamThrottleLocalReject( +function logThrottle429(state: HoldState, retryAfterHeader: string | null, now: number): void { + const streak = state.streak; + const maxC = throttle429MaxConsecutive(); + const delay = state.nextAttemptAt - now; + const raMs = parseRetryAfterMs(retryAfterHeader, now); + const raNote = raMs !== null && raMs >= delay ? ", upstream Retry-After honored" : ""; + const modelNote = state.model === undefined ? "" : ` for model "${state.model}"`; + const delaySec = Math.round(delay / 1000); + if (streak === 1) { + loggerLog("warn", `[acp-proxy] [429-hold] ${state.host}${modelNote}: upstream 429 — arming egress hold (~${delaySec}s backoff+jitter${raNote}) instead of letting host retries amplify (#4204)`); + } else if (streak === maxC) { + loggerLog("warn", `[acp-proxy] [429-hold] ${state.host}${modelNote}: ${streak} consecutive upstream 429s — hold is TERMINAL: answering locally with 424 (outside SDK retry sets) until a probe succeeds (${raNote.trim()}) (#4204)`); + } else { + loggerLog("info", `[acp-proxy] [429-hold] ${state.host}${modelNote}: consecutive 429 #${streak} — hold extended to ~${delaySec}s${raNote} (#4204)`); + } +} + +function expireProbeToken(state: HoldState, now: number): void { + if ( + state.probeToken === undefined + || state.probeAt === undefined + || now - state.probeAt < PROBE_INFLIGHT_TTL_MS + ) return; + state.probeToken = undefined; + state.probeAt = undefined; +} + +function releaseProbeToken(probeToken: symbol | undefined, now: number, accountKey?: string): void { + for (const state of holds.values()) { + if (accountKey !== undefined && state.accountKey !== accountKey) continue; + expireProbeToken(state, now); + if (probeToken === undefined || state.probeToken !== probeToken) continue; + state.probeToken = undefined; + state.probeAt = undefined; + } + for (const group of provisionalGroups.values()) { + const state = group.state; + if (accountKey !== undefined && state.accountKey !== accountKey) continue; + expireProbeToken(state, now); + if (probeToken === undefined || state.probeToken !== probeToken) continue; + state.probeToken = undefined; + state.probeAt = undefined; + } +} + +/** Release only the in-flight probe slots claimed by this failed or cancelled send. */ +export function releaseUpstreamThrottleProbe(probeToken?: symbol): void { + if (probeToken === undefined) return; + releaseProbeToken(probeToken, Date.now()); +} + +function slotBlocked(state: HoldState, now: number): boolean { + expireProbeToken(state, now); + return now < state.nextAttemptAt || state.probeToken !== undefined; +} +function strongerReject(current: HoldState | undefined, candidate: HoldState): HoldState { + if (current === undefined) return candidate; + const maxC = throttle429MaxConsecutive(); + const candidateTerminal = candidate.streak >= maxC; + const currentTerminal = current.streak >= maxC; + if (candidateTerminal && !currentTerminal) return candidate; + if (candidateTerminal === currentTerminal && candidate.nextAttemptAt > current.nextAttemptAt) return candidate; + return current; +} +function refreshProvisionalGroup(group: ProvisionalGroup): void { + const first = group.observations.values().next().value; + if (first === undefined) return; + + let strongest = first; + for (const observation of group.observations) { + strongest = strongerReject(strongest, observation); + } + + const probeToken = group.state.probeToken; + const probeAt = group.state.probeAt; + group.state = { + ...strongest, + probeToken, + probeAt, + model: undefined, + provisional: true, + provisionalRequestModel: undefined, + }; +} + +function settleProvisionalObservation( + key: string, + state: HoldState, + requestModel: string | undefined, + modelScope: string | undefined, +): void { + const exactModel = modelScope !== undefined + && modelScope.length > 0 + && requestModel !== undefined + && modelScope === requestModel + ? modelScope + : undefined; + const preserveAccountWide = exactModel !== undefined; + const targetKey = exactModel === undefined ? key : modelHoldKey(key, exactModel); + const target = holds.get(targetKey); + + state.provisional = false; + state.provisionalRequestModel = undefined; + if (exactModel === undefined) { + delete state.model; + } else { + state.model = exactModel; + } + + if (target === undefined) { + if (exactModel !== undefined) state.streak = 1; + storeHold(targetKey, state, preserveAccountWide); + } else if (target.probeToken !== state.probeToken) { + // Keep separate ownership records; a response may release only its + // own probe even when several observations converge on one scope. + state.streak = target.streak + 1; + state.nextAttemptAt = Math.max(state.nextAttemptAt, target.nextAttemptAt); + storeHold(Symbol("settled upstream throttle observation"), state, preserveAccountWide); + } else { + target.streak += 1; + target.nextAttemptAt = Math.max(target.nextAttemptAt, state.nextAttemptAt); + } +} + +/** Admit a send and return the token owning every open hold slot it claimed. */ +export function upstreamThrottleGate( url: string, headers: Record, now: number = Date.now(), -): ThrottleLocalReject | undefined { - if (!throttle429Enabled()) return undefined; + requestModel?: string, +): UpstreamThrottleGate { + if (!throttle429Enabled()) return {}; const { key } = holdKey(url, headers); - const st = holds.get(key); - if (st === undefined) return undefined; - if (now < st.nextAttemptAt) return makeReject(st, now); - // Window open: admit exactly one probe at a time; the slot self-releases - // after PROBE_INFLIGHT_TTL_MS so a never-observed verdict cannot wedge it. - if (now - st.probeAt < PROBE_INFLIGHT_TTL_MS) return makeReject(st, now); - st.probeAt = now; - return undefined; + let hasApplicable = false; + let rejectState: HoldState | undefined; + for (const state of holds.values()) { + if (state.accountKey !== key) continue; + if (!state.provisional && state.model !== undefined && state.model !== requestModel) continue; + hasApplicable = true; + if (slotBlocked(state, now)) rejectState = strongerReject(rejectState, state); + } + for (const group of provisionalGroups.values()) { + const state = group.state; + if (state.accountKey !== key) continue; + hasApplicable = true; + if (slotBlocked(state, now)) rejectState = strongerReject(rejectState, state); + } + if (rejectState !== undefined) return { reject: makeReject(rejectState, now) }; + if (!hasApplicable) return {}; + + const probeToken = Symbol("upstream throttle probe"); + for (const state of holds.values()) { + if (state.accountKey !== key) continue; + if (!state.provisional && state.model !== undefined && state.model !== requestModel) continue; + state.probeToken = probeToken; + state.probeAt = now; + } + for (const group of provisionalGroups.values()) { + const state = group.state; + if (state.accountKey !== key) continue; + state.probeToken = probeToken; + state.probeAt = now; + } + return { probeToken }; } -/** Observe an actual upstream 429 verdict for this send. Arms/extends the - * hold: exponential backoff ×2 per consecutive 429, jittered into [0.5×, 1×), - * capped; an upstream Retry-After (seconds or HTTP-date) wins when larger. */ + +/** Observe a classified upstream 429 verdict for this send. */ export function noteUpstreamThrottle429( url: string, headers: Record, retryAfterHeader: string | null, now: number = Date.now(), + modelScope?: string, + probeToken?: symbol, ): void { if (!throttle429Enabled()) return; const { key, host, cred } = holdKey(url, headers); - const prev = holds.get(key); - const streak = (prev?.streak ?? 0) + 1; - const maxC = throttle429MaxConsecutive(); - const base = throttle429BaseDelayMs(); - const cap = throttle429MaxDelayMs(); - const exp = Math.min(cap, base * 2 ** Math.min(streak - 1, 30)); - const jittered = exp * (0.5 + Math.random() * 0.5); - const raMs = parseRetryAfterMs(retryAfterHeader, now); - const delay = Math.max(jittered, raMs ?? 0); - const nextAttemptAt = now + delay; - if (!prev && holds.size >= MAX_HOLDS) evictOldest(); - holds.set(key, { streak, nextAttemptAt, probeAt: 0, host, cred }); - const delaySec = Math.round(delay / 1000); - const raNote = raMs !== null && raMs >= jittered ? ", upstream Retry-After honored" : ""; - if (streak === 1) { - loggerLog("warn", `[acp-proxy] [429-hold] ${host}: upstream 429 — arming egress hold (~${delaySec}s backoff+jitter${raNote}) instead of letting host retries amplify (#4204)`); - } else if (streak === maxC) { - loggerLog("warn", `[acp-proxy] [429-hold] ${host}: ${streak} consecutive upstream 429s — hold is TERMINAL: answering locally with 424 (outside SDK retry sets) until a probe succeeds (${raNote.trim()}) (#4204)`); + releaseProbeToken(probeToken, now, key); + const scopeModel = modelScope === undefined || modelScope.length === 0 ? undefined : modelScope; + const stateKey = scopeModel === undefined ? key : modelHoldKey(key, scopeModel); + const previous = holds.get(stateKey); + const observedState = createHoldState(key, host, cred, retryAfterHeader, now, previous, scopeModel, false); + if (previous === undefined) { + if (!storeHold(stateKey, observedState, scopeModel !== undefined)) return; } else { - loggerLog("info", `[acp-proxy] [429-hold] ${host}: consecutive 429 #${streak} — hold extended to ~${delaySec}s${raNote} (#4204)`); + holds.set(stateKey, observedState); } + logThrottle429(observedState, retryAfterHeader, now); } -/** Observe ANY resolved non-429 verdict for this send (2xx included): the - * endpoint is reachable and processing again — clear the hold immediately. */ +/** + * Arm an account-scoped provisional observation while a small response body is + * still being classified. Each returned settler owns exactly this observation: + * an exact request-model match narrows it, while every other result confirms + * account scope without discarding concurrent or already-verified evidence. + * A supplied classification reservation is released by this observation's settler. + * Legacy callers that omit it acquire a reservation internally. + */ +export function beginProvisionalUpstreamThrottle429( + url: string, + headers: Record, + retryAfterHeader: string | null, + now: number, + requestModel: string | undefined, + probeToken?: symbol, + reservation?: ThrottleClassificationReservation, +): (modelScope?: string) => void { + if (!throttle429Enabled()) { + reservation?.release(); + return () => undefined; + } + const { key, host, cred } = holdKey(url, headers); + releaseProbeToken(probeToken, now, key); + + const classificationReservation = reservation ?? reserveUpstreamThrottleClassification(url, headers); + if (classificationReservation === undefined) return () => undefined; + const classificationState = classificationReservations.get(classificationReservation); + if (classificationState === undefined || !classificationState.active || classificationState.claimed) { + return () => undefined; + } + if (classificationState.accountKey !== key) { + classificationReservation.release(); + return () => undefined; + } + classificationState.claimed = true; + + const previousAccount = holds.get(key); + const state = createHoldState( + key, + host, + cred, + retryAfterHeader, + now, + previousAccount, + undefined, + true, + requestModel, + ); + // This observation has completed its send; it must not inherit ownership of + // an earlier account probe merely because it was based on that backoff. + state.probeToken = undefined; + state.probeAt = undefined; + + const existingGroup = provisionalGroups.get(key); + if (existingGroup === undefined && provisionalGroups.size >= MAX_CLASSIFICATION_ACCOUNTS) { + classificationReservation.release(); + return () => undefined; + } + const group = existingGroup ?? { + state: { ...state, provisionalRequestModel: undefined }, + observations: new Set(), + }; + if (existingGroup === undefined) provisionalGroups.set(key, group); + if (group.observations.size >= MAX_CLASSIFICATIONS_PER_ACCOUNT) { + classificationReservation.release(); + return () => undefined; + } + + group.observations.add(state); + refreshProvisionalGroup(group); + logThrottle429(state, retryAfterHeader, now); + + classificationState.onRelease = () => { + if (provisionalGroups.get(key) !== group || !group.observations.delete(state)) return; + if (group.observations.size === 0) { + provisionalGroups.delete(key); + } else { + refreshProvisionalGroup(group); + } + }; + + let settled = false; + return (modelScope?: string): void => { + if (settled) return; + settled = true; + const pendingGroup = provisionalGroups.get(key); + if (pendingGroup !== group || !group.observations.delete(state)) { + classificationReservation.release(); + return; + } + classificationState.onRelease = undefined; + settleProvisionalObservation(key, state, requestModel, modelScope); + + if (group.observations.size === 0) { + provisionalGroups.delete(key); + } else { + refreshProvisionalGroup(group); + } + classificationReservation.release(); + }; +} + +/** A resolved non-429 clears confirmed account and matching-model holds. */ export function noteUpstreamThrottleResolved( url: string, headers: Record, status: number, now: number = Date.now(), + requestModel?: string, + probeToken?: symbol, ): void { - void now; if (!throttle429Enabled()) return; const { key, host } = holdKey(url, headers); - const st = holds.get(key); - if (st === undefined) return; - holds.delete(key); - loggerLog("info", `[acp-proxy] [429-hold] ${host}: cleared after HTTP ${status} (was ${st.streak} consecutive 429s) (#4204)`); + + for (const [slotKey, state] of holds) { + if (state.accountKey !== key) continue; + expireProbeToken(state, now); + const ownsProbe = probeToken !== undefined && state.probeToken === probeToken; + const inResolvedScope = state.model === undefined || state.model === requestModel; + + // A completed send releases only the slots its admission token owns. + // Provisional observations are settled by their own body reader, not by + // an unrelated response that happens to have the same request model. + if (state.provisional) { + if (ownsProbe) { + state.probeToken = undefined; + state.probeAt = undefined; + } + continue; + } + if (!inResolvedScope) { + if (ownsProbe) { + state.probeToken = undefined; + state.probeAt = undefined; + } + continue; + } + if (state.probeToken !== undefined && !ownsProbe) continue; + + holds.delete(slotKey); + const modelNote = state.model === undefined ? "" : ` for model "${state.model}"`; + loggerLog("info", `[acp-proxy] [429-hold] ${host}${modelNote}: cleared after HTTP ${status} (was ${state.streak} consecutive 429s) (#4204)`); + } + for (const group of provisionalGroups.values()) { + const state = group.state; + if (state.accountKey !== key) continue; + expireProbeToken(state, now); + if (probeToken !== undefined && state.probeToken === probeToken) { + state.probeToken = undefined; + state.probeAt = undefined; + } + } } -/** Display surface for acp_status / web overview (mirrors getOpenUpstreamCircuits). */ interface ActiveThrottleHold { host: string; streak: number; nextAttemptAt: number; terminal: boolean; + model?: string; } export function activeThrottleHolds(): ActiveThrottleHold[] { const maxC = throttle429MaxConsecutive(); - return [...holds.values()].map((st) => ({ - host: st.host, - streak: st.streak, - nextAttemptAt: st.nextAttemptAt, - terminal: st.streak >= maxC, + const states = [ + ...holds.values(), + ...Array.from(provisionalGroups.values(), (group) => group.state), + ]; + return states.map((state) => ({ + host: state.host, + streak: state.streak, + nextAttemptAt: state.nextAttemptAt, + terminal: state.streak >= maxC, + ...(state.model === undefined ? {} : { model: state.model }), })); } export function _resetUpstreamThrottleForTest(): void { holds.clear(); + provisionalGroups.clear(); + for (const state of classificationReservations.values()) { + state.active = false; + state.onRelease = undefined; + } + classificationReservations.clear(); + activeClassificationCounts.clear(); } diff --git a/tests/issue4204-throttle-governor.test.ts b/tests/issue4204-throttle-governor.test.ts index 5d333f2cf4..b78cde6a44 100644 --- a/tests/issue4204-throttle-governor.test.ts +++ b/tests/issue4204-throttle-governor.test.ts @@ -1,25 +1,32 @@ -// #4204: egress-side 429 throttle governor — consecutive upstream 429s stop -// amplifying: local 429+Retry-After holds (exponential backoff × jitter, -// upstream Retry-After honored), TERMINAL 424 outside SDK retry sets once the -// budget is exhausted, one probe per backoff window for automatic recovery, -// ANY resolved non-429 clears immediately, per origin+credential isolation. +// #4204: egress-side 429 governor — consecutive upstream 429s stop amplifying: +// account limits hold origin+credential; a request-matched KeyRing model +// refusal holds only origin+credential+model. import { test } from "node:test"; import assert from "node:assert/strict"; import { activeThrottleHolds, + beginProvisionalUpstreamThrottle429, + keyringModel429Scope, noteUpstreamThrottle429, noteUpstreamThrottleResolved, - upstreamThrottleLocalReject, + requestModelFromBody, + reserveUpstreamThrottleClassification, + upstreamThrottleGate, _resetUpstreamThrottleForTest, } from "../src/upstream-throttle.ts"; const URL_A = "http://z.example/api/coding/paas/v4/chat/completions"; +const URL_B = "http://keyring-b.example/api/coding/paas/v4/chat/completions"; const H_EMPTY: Record = {}; const H_CRED_A: Record = { authorization: "Bearer AAA" }; const H_CRED_B: Record = { authorization: "Bearer BBB" }; // Integer-second timestamps: keeps HTTP-date Retry-After fixtures exact. const T0 = 1_000_000; +function throttleReject(url: string, headers: Record, now?: number, requestModel?: string) { + return upstreamThrottleGate(url, headers, now, requestModel).reject; +} + const KNOB_ENVS = [ "BILI_THROTTLE_429", "BILI_THROTTLE_429_MAX_CONSECUTIVE", @@ -43,11 +50,39 @@ function withKnobs(vars: Partial>, fn } } +test("#4204: request model extraction is not limited by the error-body cap", () => { + const model = "deepseek-v4.1-flash"; + const body = JSON.stringify({ model, messages: [{ content: "x".repeat(20 * 1024) }] }); + assert.ok(new TextEncoder().encode(body).byteLength > 16 * 1024); + assert.equal(requestModelFromBody(body), model); + assert.equal(requestModelFromBody(new TextEncoder().encode(body)), model); +}); +test("#4204: passthrough model scan uses top-level JSON semantics without parsing nested history", () => { + assert.equal( + requestModelFromBody('{"metadata":{"model":"nested"},"\\u006dodel":"escaped"}'), + "escaped", + "escaped top-level key is recognized while nested model keys are ignored", + ); + assert.equal(requestModelFromBody('{"model":"first","model":"last"}'), "last", "duplicate members follow JSON.parse last-value semantics"); + assert.equal(requestModelFromBody('{"model":"first","model":1}'), undefined, "a non-string last member clears an earlier model"); +}); + +test("#4204: Bili local model hold marker requires its exact scope and request model", () => { + const model = "deepseek-v4.1-flash"; + const marker = (scope: string | undefined, responseModel: string) => + JSON.stringify({ error: { code: "bili_throttle_hold", ...(scope === undefined ? {} : { scope }), model: responseModel } }); + assert.equal(keyringModel429Scope(429, model, marker("model", model)), model); + assert.equal(keyringModel429Scope(429, model, marker("account", model)), undefined); + assert.equal(keyringModel429Scope(429, model, marker(undefined, model)), undefined); + assert.equal(keyringModel429Scope(429, model, marker("model", "different-model")), undefined); + assert.equal(keyringModel429Scope(400, model, marker("model", model)), undefined); +}); + test("#4204: fresh origin is allowed; nothing armed yet (negative row)", () => { withKnobs({ BILI_THROTTLE_429: "1" }, () => { _resetUpstreamThrottleForTest(); assert.equal(activeThrottleHolds().length, 0); - assert.equal(upstreamThrottleLocalReject(URL_A, H_EMPTY, T0), undefined); + assert.equal(throttleReject(URL_A, H_EMPTY, T0), undefined); }); }); @@ -62,7 +97,7 @@ test("#4204: first 429 arms a hold — next send answered locally 429 + Retry-Af // streak 1: exp = base → jittered delay lands in [base/2, base]. assert.ok(st.nextAttemptAt - T0 >= 1000 && st.nextAttemptAt - T0 <= 2000, `delay ${st.nextAttemptAt - T0} outside [1000,2000]`); - const g = upstreamThrottleLocalReject(URL_A, H_EMPTY, T0 + 100); + const g = throttleReject(URL_A, H_EMPTY, T0 + 100); assert.ok(g, "send during the window must be rejected locally"); assert.equal(g.status, 429); assert.equal(g.code, "bili_throttle_hold"); @@ -73,7 +108,7 @@ test("#4204: first 429 arms a hold — next send answered locally 429 + Retry-Af assert.equal(g.maxConsecutive, 5); // Repeated local rejects must not mutate the hold (no state drift). - const g2 = upstreamThrottleLocalReject(URL_A, H_EMPTY, T0 + 200); + const g2 = throttleReject(URL_A, H_EMPTY, T0 + 200); assert.ok(g2); assert.equal(g2.nextAttemptAt, st.nextAttemptAt); assert.equal(g2.streak, 1); @@ -84,10 +119,10 @@ test("#4204: any resolved non-429 clears the hold (400 counts — endpoint is pr withKnobs({ BILI_THROTTLE_429: "1", BILI_THROTTLE_429_MAX_CONSECUTIVE: "5", BILI_THROTTLE_429_BASE_DELAY_MS: "2000", BILI_THROTTLE_429_MAX_DELAY_MS: "30000" }, () => { _resetUpstreamThrottleForTest(); noteUpstreamThrottle429(URL_A, H_EMPTY, null, T0); - assert.ok(upstreamThrottleLocalReject(URL_A, H_EMPTY, T0 + 100), "armed: rejected"); + assert.ok(throttleReject(URL_A, H_EMPTY, T0 + 100), "armed: rejected"); noteUpstreamThrottleResolved(URL_A, H_EMPTY, 400); assert.equal(activeThrottleHolds().length, 0, "hold cleared"); - assert.equal(upstreamThrottleLocalReject(URL_A, H_EMPTY, T0 + 200), undefined, "allowed again"); + assert.equal(throttleReject(URL_A, H_EMPTY, T0 + 200), undefined, "allowed again"); }); }); @@ -102,7 +137,7 @@ test("#4204: budget exhausted → TERMINAL 424 without Retry-After; probe then s assert.equal(st.terminal, true); // Terminal verdict holds even mid-window — hosts must give up now. - const g = upstreamThrottleLocalReject(URL_A, H_EMPTY, T0 + 1000); + const g = throttleReject(URL_A, H_EMPTY, T0 + 1000); assert.ok(g); assert.equal(g.status, 424); assert.equal(g.code, "bili_throttle_terminal"); @@ -110,11 +145,12 @@ test("#4204: budget exhausted → TERMINAL 424 without Retry-After; probe then s assert.equal(g.streak, 3); assert.equal(g.maxConsecutive, 3); - // Window opens → exactly one probe admitted → success clears everything. - assert.equal(upstreamThrottleLocalReject(URL_A, H_EMPTY, st.nextAttemptAt), undefined, "probe admitted at window edge"); - noteUpstreamThrottleResolved(URL_A, H_EMPTY, 200); + const probe = upstreamThrottleGate(URL_A, H_EMPTY, st.nextAttemptAt); + assert.equal(probe.reject, undefined, "window opens and admits a probe"); + assert.equal(typeof probe.probeToken, "symbol", "admitted probe has a request-owned token"); + noteUpstreamThrottleResolved(URL_A, H_EMPTY, 200, st.nextAttemptAt + 1, undefined, probe.probeToken); assert.equal(activeThrottleHolds().length, 0, "cleared after successful probe"); - assert.equal(upstreamThrottleLocalReject(URL_A, H_EMPTY, st.nextAttemptAt + 1), undefined); + assert.equal(upstreamThrottleGate(URL_A, H_EMPTY, st.nextAttemptAt + 2).reject, undefined); }); }); @@ -159,10 +195,16 @@ test("#4204: single-flight probe — one admit per open window, slot self-releas _resetUpstreamThrottleForTest(); noteUpstreamThrottle429(URL_A, H_EMPTY, null, T0); const edge = activeThrottleHolds()[0].nextAttemptAt; - assert.equal(upstreamThrottleLocalReject(URL_A, H_EMPTY, edge), undefined, "first caller at window edge gets the probe"); - const dup = upstreamThrottleLocalReject(URL_A, H_EMPTY, edge); - assert.ok(dup, "concurrent caller while probe is in flight is held locally"); - assert.equal(upstreamThrottleLocalReject(URL_A, H_EMPTY, edge + 30_000), undefined, "slot self-releases after the probe TTL even without a verdict"); + const first = upstreamThrottleGate(URL_A, H_EMPTY, edge); + assert.equal(first.reject, undefined, "first caller at window edge gets the probe"); + assert.equal(typeof first.probeToken, "symbol"); + const dup = upstreamThrottleGate(URL_A, H_EMPTY, edge); + assert.ok(dup.reject, "concurrent caller while probe is in flight is held locally"); + assert.equal(dup.probeToken, undefined, "rejected sends do not receive a token"); + const afterTtl = upstreamThrottleGate(URL_A, H_EMPTY, edge + 30_000); + assert.equal(afterTtl.reject, undefined, "slot self-releases after the probe TTL even without a verdict"); + assert.equal(typeof afterTtl.probeToken, "symbol"); + assert.notEqual(afterTtl.probeToken, first.probeToken, "each admitted probe has unique ownership"); }); }); @@ -170,20 +212,381 @@ test("#4204: same origin, different credentials — independent pools, no cross- withKnobs({ BILI_THROTTLE_429: "1", BILI_THROTTLE_429_MAX_CONSECUTIVE: "5", BILI_THROTTLE_429_BASE_DELAY_MS: "2000", BILI_THROTTLE_429_MAX_DELAY_MS: "30000" }, () => { _resetUpstreamThrottleForTest(); noteUpstreamThrottle429(URL_A, H_CRED_A, null, T0); - assert.ok(upstreamThrottleLocalReject(URL_A, H_CRED_A, T0 + 100), "credential A held"); - assert.equal(upstreamThrottleLocalReject(URL_A, H_CRED_B, T0 + 100), undefined, "credential B unaffected"); - assert.equal(upstreamThrottleLocalReject(URL_A, H_EMPTY, T0 + 100), undefined, "credential-less lane unaffected"); + assert.ok(throttleReject(URL_A, H_CRED_A, T0 + 100), "credential A held"); + assert.equal(throttleReject(URL_A, H_CRED_B, T0 + 100), undefined, "credential B unaffected"); + assert.equal(throttleReject(URL_A, H_EMPTY, T0 + 100), undefined, "credential-less lane unaffected"); assert.equal(activeThrottleHolds().length, 1, "only one pool armed"); }); }); +test("#4204: a matching KeyRing model refusal holds only that model; account quota remains origin+credential", () => { + withKnobs({ BILI_THROTTLE_429: "1", BILI_THROTTLE_429_MAX_CONSECUTIVE: "5", BILI_THROTTLE_429_BASE_DELAY_MS: "1000", BILI_THROTTLE_429_MAX_DELAY_MS: "8000" }, () => { + _resetUpstreamThrottleForTest(); + const model = "deepseek-v4.1-flash"; + const otherModel = "qwen3.8-flash"; + const modelError = JSON.stringify({ + error: { code: "keyring_model_temporarily_unavailable", model }, + }); + const modelScope = keyringModel429Scope(429, model, modelError); + assert.equal(modelScope, model, "only the exact KeyRing model code+request model grants model scope"); + assert.equal(keyringModel429Scope(429, otherModel, modelError), undefined, "response model must match request model"); + assert.equal( + keyringModel429Scope(429, model, JSON.stringify({ error: { code: "keyring_all_limited", model } })), + undefined, + "keyring_all_limited is account quota even when it echoes a model", + ); + assert.equal( + keyringModel429Scope( + 429, + model, + JSON.stringify({ error: { code: "GoUsageLimitError", message: "monthly OpenCode Go usage limit exhausted" } }), + ), + undefined, + "direct OpenCode monthly quota is not a KeyRing model-local marker", + ); + assert.equal(keyringModel429Scope(429, model, "{malformed"), undefined); + assert.equal(keyringModel429Scope(400, model, modelError), undefined); + + noteUpstreamThrottle429(URL_A, H_CRED_A, null, T0, modelScope); + assert.ok(throttleReject(URL_A, H_CRED_A, T0 + 100, model), "the refused model is held"); + assert.equal(throttleReject(URL_A, H_CRED_A, T0 + 100, otherModel), undefined, "another model on the same credential stays eligible"); + assert.equal(activeThrottleHolds()[0].model, model); + noteUpstreamThrottleResolved(URL_A, H_CRED_A, 200, T0 + 200, otherModel); + assert.ok(throttleReject(URL_A, H_CRED_A, T0 + 300, model), "another model's success does not clear this model hold"); + noteUpstreamThrottleResolved(URL_A, H_CRED_A, 200, T0 + 400, model); + assert.equal(throttleReject(URL_A, H_CRED_A, T0 + 500, model), undefined, "the matching model's resolved response clears its hold"); + assert.equal(activeThrottleHolds().length, 0); + + const accountError = JSON.stringify({ error: { code: "GoUsageLimitError", message: "monthly OpenCode Go usage limit exceeded" } }); + const accountScope = keyringModel429Scope(429, model, accountError); + assert.equal(accountScope, undefined); + noteUpstreamThrottle429(URL_A, H_CRED_A, null, T0 + 600, accountScope); + assert.equal(activeThrottleHolds()[0].model, undefined, "account quota is not assigned a model scope"); + assert.ok(throttleReject(URL_A, H_CRED_A, T0 + 700, model), "account quota holds the requested model"); + assert.ok(throttleReject(URL_A, H_CRED_A, T0 + 700, otherModel), "account quota holds every model on this origin+credential"); + _resetUpstreamThrottleForTest(); + }); +}); + +test("#4204: overlapping account and model holds select the strongest reject and share one probe", () => { + withKnobs({ BILI_THROTTLE_429: "1", BILI_THROTTLE_429_MAX_CONSECUTIVE: "5", BILI_THROTTLE_429_BASE_DELAY_MS: "1000", BILI_THROTTLE_429_MAX_DELAY_MS: "8000" }, () => { + _resetUpstreamThrottleForTest(); + const model = "deepseek-v4.1-flash"; + noteUpstreamThrottle429(URL_A, H_CRED_A, "10", T0, model); + noteUpstreamThrottle429(URL_A, H_CRED_A, "5", T0 + 1); + const modelHold = activeThrottleHolds().find((hold) => hold.model === model); + const accountHold = activeThrottleHolds().find((hold) => hold.model === undefined); + assert.ok(modelHold); + assert.ok(accountHold); + const edge = Math.max(modelHold.nextAttemptAt, accountHold.nextAttemptAt); + + const strongest = throttleReject(URL_A, H_CRED_A, T0 + 100, model); + assert.ok(strongest); + assert.equal(strongest.model, model, "the applicable hold with the longest backoff is selected"); + assert.equal(throttleReject(URL_A, H_CRED_A, T0 + 100, "other-model")?.model, undefined); + + const probe = upstreamThrottleGate(URL_A, H_CRED_A, edge, model); + assert.equal(probe.reject, undefined, "one request claims both open probe slots"); + assert.equal(typeof probe.probeToken, "symbol"); + assert.ok(upstreamThrottleGate(URL_A, H_CRED_A, edge + 1, model).reject, "the overlapping model hold is still in flight"); + assert.ok(upstreamThrottleGate(URL_A, H_CRED_A, edge + 1, "other-model").reject, "the account hold shares the same probe slot"); + noteUpstreamThrottleResolved(URL_A, H_CRED_A, 200, edge + 2, model, probe.probeToken); + assert.equal(activeThrottleHolds().length, 0); + }); +}); + +test("#4204: a 429 releases only the request-owned probe even when observation scope differs", () => { + withKnobs({ BILI_THROTTLE_429: "1", BILI_THROTTLE_429_MAX_CONSECUTIVE: "5", BILI_THROTTLE_429_BASE_DELAY_MS: "1000", BILI_THROTTLE_429_MAX_DELAY_MS: "8000" }, () => { + _resetUpstreamThrottleForTest(); + const requestModel = "request-model"; + const observedScope = "different-observation-scope"; + noteUpstreamThrottle429(URL_A, H_CRED_A, null, T0, requestModel); + const edge = activeThrottleHolds()[0].nextAttemptAt; + const probe = upstreamThrottleGate(URL_A, H_CRED_A, edge, requestModel); + assert.equal(probe.reject, undefined, "the original model probe is claimed"); + assert.ok(upstreamThrottleGate(URL_A, H_CRED_A, edge + 1, requestModel).reject, "the probe starts in flight"); + + noteUpstreamThrottle429(URL_A, H_CRED_A, null, edge + 1, observedScope, probe.probeToken); + const next = upstreamThrottleGate(URL_A, H_CRED_A, edge + 2, requestModel); + assert.equal(next.reject, undefined, "the actual request-model slot is released despite the changed observation scope"); + assert.equal(typeof next.probeToken, "symbol", "the original model hold admits a new probe after its owner completes"); + assert.notEqual(next.probeToken, probe.probeToken, "a later probe receives distinct ownership"); + }); +}); +test("#4204: a matching-model success cannot release another request's probe", () => { + withKnobs({ BILI_THROTTLE_429: "1", BILI_THROTTLE_429_MAX_CONSECUTIVE: "5", BILI_THROTTLE_429_BASE_DELAY_MS: "1000", BILI_THROTTLE_429_MAX_DELAY_MS: "8000" }, () => { + _resetUpstreamThrottleForTest(); + const model = "request-model"; + noteUpstreamThrottle429(URL_A, H_CRED_A, null, T0, model); + const edge = activeThrottleHolds()[0].nextAttemptAt; + const probe = upstreamThrottleGate(URL_A, H_CRED_A, edge, model); + assert.equal(probe.reject, undefined); + assert.ok(probe.probeToken); + + noteUpstreamThrottleResolved(URL_A, H_CRED_A, 200, edge + 1, model, Symbol("unrelated request")); + assert.ok(upstreamThrottleGate(URL_A, H_CRED_A, edge + 2, model).reject, "model equality alone cannot release an active probe"); + + noteUpstreamThrottleResolved(URL_A, H_CRED_A, 200, edge + 3, model, probe.probeToken); + assert.equal(activeThrottleHolds().length, 0, "the owning completion clears its model hold"); + }); +}); +test("#4204: concurrent provisional observations settle independently", () => { + withKnobs({ BILI_THROTTLE_429: "1", BILI_THROTTLE_429_MAX_CONSECUTIVE: "5", BILI_THROTTLE_429_BASE_DELAY_MS: "1000", BILI_THROTTLE_429_MAX_DELAY_MS: "8000" }, () => { + _resetUpstreamThrottleForTest(); + const modelA = "model-a"; + const modelB = "model-b"; + const settleA = beginProvisionalUpstreamThrottle429(URL_A, H_CRED_A, null, T0, modelA); + const settleB = beginProvisionalUpstreamThrottle429(URL_A, H_CRED_A, null, T0 + 1, modelB); + + assert.ok(upstreamThrottleGate(URL_A, H_CRED_A, T0 + 2, "unrelated-model").reject, "unresolved scope is conservatively account-wide"); + settleA(modelA); + assert.ok(activeThrottleHolds().some((hold) => hold.model === modelA), "first observation narrows to model A"); + assert.ok(activeThrottleHolds().some((hold) => hold.model === undefined), "second pending observation remains account-scoped"); + assert.ok(upstreamThrottleGate(URL_A, H_CRED_A, T0 + 3, modelA).reject, "the other pending observation still gates model A"); + assert.ok(upstreamThrottleGate(URL_A, H_CRED_A, T0 + 3, modelB).reject, "the other pending observation still gates model B"); + + settleB(modelB); + const refined = activeThrottleHolds(); + assert.equal(refined.length, 2); + assert.ok(refined.some((hold) => hold.model === modelA)); + assert.ok(refined.some((hold) => hold.model === modelB)); + assert.equal(upstreamThrottleGate(URL_A, H_CRED_A, T0 + 4, "unrelated-model").reject, undefined, "independent exact refinements remove only their account-wide provisional holds"); + }); +}); +test("#4204: an ambiguous concurrent refinement remains account-wide beside exact model evidence", () => { + withKnobs({ BILI_THROTTLE_429: "1", BILI_THROTTLE_429_MAX_CONSECUTIVE: "5", BILI_THROTTLE_429_BASE_DELAY_MS: "1000", BILI_THROTTLE_429_MAX_DELAY_MS: "8000" }, () => { + _resetUpstreamThrottleForTest(); + const modelA = "model-a"; + const settleExact = beginProvisionalUpstreamThrottle429(URL_A, H_CRED_A, null, T0, modelA); + const settleAmbiguous = beginProvisionalUpstreamThrottle429(URL_A, H_CRED_A, null, T0 + 1, "model-b"); + settleExact(modelA); + settleAmbiguous(undefined); + + assert.ok(activeThrottleHolds().some((hold) => hold.model === modelA), "exact evidence is model-scoped"); + assert.ok(activeThrottleHolds().some((hold) => hold.model === undefined), "ambiguous evidence remains account-scoped"); + assert.ok(upstreamThrottleGate(URL_A, H_CRED_A, T0 + 2, "other-model").reject); + }); +}); +test("#4204: exact refinement preserves genuine account evidence", () => { + withKnobs({ BILI_THROTTLE_429: "1", BILI_THROTTLE_429_MAX_CONSECUTIVE: "5", BILI_THROTTLE_429_BASE_DELAY_MS: "1000", BILI_THROTTLE_429_MAX_DELAY_MS: "8000" }, () => { + _resetUpstreamThrottleForTest(); + const model = "deepseek-v4.1-flash"; + noteUpstreamThrottle429(URL_A, H_CRED_A, null, T0); + const settle = beginProvisionalUpstreamThrottle429(URL_A, H_CRED_A, null, T0 + 1, model); + settle(model); + + assert.ok(activeThrottleHolds().some((hold) => hold.model === undefined), "verified account evidence remains account-wide"); + assert.ok(activeThrottleHolds().some((hold) => hold.model === model), "the provisional observation becomes a separate model hold"); + assert.ok(upstreamThrottleGate(URL_A, H_CRED_A, T0 + 2, "other-model").reject, "the verified account hold still gates other models"); + }); +}); +test("#4204: ambiguous provisional settlement confirms account scope", () => { + withKnobs({ BILI_THROTTLE_429: "1", BILI_THROTTLE_429_MAX_CONSECUTIVE: "5", BILI_THROTTLE_429_BASE_DELAY_MS: "1000", BILI_THROTTLE_429_MAX_DELAY_MS: "8000" }, () => { + _resetUpstreamThrottleForTest(); + const settle = beginProvisionalUpstreamThrottle429(URL_A, H_CRED_A, null, T0, "request-model"); + settle(undefined); + assert.equal(activeThrottleHolds().length, 1); + assert.equal(activeThrottleHolds()[0].model, undefined); + assert.ok(upstreamThrottleGate(URL_A, H_CRED_A, T0 + 1, "other-model").reject); + }); +}); +test("#4204: an unrelated success does not consume a pending refinement", () => { + withKnobs({ BILI_THROTTLE_429: "1", BILI_THROTTLE_429_MAX_CONSECUTIVE: "5", BILI_THROTTLE_429_BASE_DELAY_MS: "1000", BILI_THROTTLE_429_MAX_DELAY_MS: "8000" }, () => { + _resetUpstreamThrottleForTest(); + const model = "deepseek-v4.1-flash"; + const settle = beginProvisionalUpstreamThrottle429(URL_A, H_CRED_A, null, T0, model); + noteUpstreamThrottleResolved(URL_A, H_CRED_A, 200, T0 + 1, "other-model"); + settle(model); + assert.equal(activeThrottleHolds().length, 1); + assert.equal(activeThrottleHolds()[0].model, model, "only the observation's own settlement refines it"); + assert.ok(upstreamThrottleGate(URL_A, H_CRED_A, T0 + 2, model).reject); + assert.equal(upstreamThrottleGate(URL_A, H_CRED_A, T0 + 2, "other-model").reject, undefined); + _resetUpstreamThrottleForTest(); + }); +}); + +test("#4204: provisional settlement is one-shot and only an exact request model narrows", () => { + withKnobs({ BILI_THROTTLE_429: "1", BILI_THROTTLE_429_MAX_CONSECUTIVE: "5", BILI_THROTTLE_429_BASE_DELAY_MS: "1000", BILI_THROTTLE_429_MAX_DELAY_MS: "8000" }, () => { + _resetUpstreamThrottleForTest(); + const model = "deepseek-v4.1-flash"; + const settle = beginProvisionalUpstreamThrottle429(URL_A, H_CRED_A, null, T0, model); + settle("different-model"); + settle(model); + assert.equal(activeThrottleHolds().length, 1); + assert.equal(activeThrottleHolds()[0].model, undefined, "a mismatch confirms account scope and later calls cannot narrow it"); + assert.ok(upstreamThrottleGate(URL_A, H_CRED_A, T0 + 1, "other-model").reject); + _resetUpstreamThrottleForTest(); + }); +}); + +test("#4204: 64 pending observations share bounded provisional state without evicting account holds", () => { + withKnobs({ BILI_THROTTLE_429: "1", BILI_THROTTLE_429_MAX_CONSECUTIVE: "5", BILI_THROTTLE_429_BASE_DELAY_MS: "1000", BILI_THROTTLE_429_MAX_DELAY_MS: "8000" }, () => { + _resetUpstreamThrottleForTest(); + noteUpstreamThrottle429(URL_A, H_CRED_A, null, T0); + const model = "model-b"; + const settlers: Array<(modelScope?: string) => void> = []; + for (let i = 0; i < 64; i++) { + settlers.push(beginProvisionalUpstreamThrottle429(URL_B, H_CRED_B, null, T0 + i, model)); + } + + const pending = activeThrottleHolds(); + assert.equal(pending.length, 2, "64 observations on B share one account-scoped pending gate"); + assert.ok(pending.some((hold) => hold.host === "z.example" && hold.model === undefined), "A's confirmed account hold remains recorded"); + assert.ok(pending.some((hold) => hold.host === "keyring-b.example" && hold.model === undefined), "B remains conservatively account-scoped while observations are pending"); + assert.ok(throttleReject(URL_A, H_CRED_A, T0 + 64), "A remains locally blocked"); + assert.ok(throttleReject(URL_B, H_CRED_B, T0 + 64, "other-model"), "pending B evidence gates unrelated models"); + + settlers[0](model); + assert.ok(throttleReject(URL_B, H_CRED_B, T0 + 65, "other-model"), "one exact settlement does not consume the other pending observations"); + for (let i = 1; i < settlers.length; i++) settlers[i](model); + + const settled = activeThrottleHolds(); + assert.equal(settled.length, 2, "settled observations converge onto one model-local B hold"); + assert.ok(settled.some((hold) => hold.host === "z.example" && hold.model === undefined), "B's exact settlements do not alter A"); + assert.ok(settled.some((hold) => hold.host === "keyring-b.example" && hold.model === model), "B is held only for its verified model"); + assert.ok(throttleReject(URL_A, H_CRED_A, T0 + 100), "A remains locally blocked after every B settlement"); + assert.ok(throttleReject(URL_B, H_CRED_B, T0 + 100, model), "the verified B model remains locally blocked"); + assert.equal(throttleReject(URL_B, H_CRED_B, T0 + 100, "other-model"), undefined, "unrelated B models become eligible"); + }); +}); + +test("#4204: ambiguous B settlement preserves B account scope beside confirmed A", () => { + withKnobs({ BILI_THROTTLE_429: "1", BILI_THROTTLE_429_MAX_CONSECUTIVE: "5", BILI_THROTTLE_429_BASE_DELAY_MS: "1000", BILI_THROTTLE_429_MAX_DELAY_MS: "8000" }, () => { + _resetUpstreamThrottleForTest(); + noteUpstreamThrottle429(URL_A, H_CRED_A, null, T0); + const settleB = beginProvisionalUpstreamThrottle429(URL_B, H_CRED_B, null, T0 + 1, "model-b"); + settleB(undefined); + + const holds = activeThrottleHolds(); + assert.ok(holds.some((hold) => hold.host === "z.example" && hold.model === undefined), "A's confirmed account hold remains recorded"); + assert.ok(holds.some((hold) => hold.host === "keyring-b.example" && hold.model === undefined), "ambiguous B evidence stays account-wide"); + assert.ok(throttleReject(URL_A, H_CRED_A, T0 + 2), "A remains locally blocked"); + assert.ok(throttleReject(URL_B, H_CRED_B, T0 + 2, "other-model"), "ambiguous B evidence gates every model"); + }); +}); + +test("#4204: model-local inserts at capacity do not evict confirmed account holds", () => { + withKnobs({ BILI_THROTTLE_429: "1", BILI_THROTTLE_429_MAX_CONSECUTIVE: "5", BILI_THROTTLE_429_BASE_DELAY_MS: "1000", BILI_THROTTLE_429_MAX_DELAY_MS: "8000" }, () => { + _resetUpstreamThrottleForTest(); + noteUpstreamThrottle429(URL_A, H_CRED_A, null, T0); + for (let i = 0; i < 63; i++) { + noteUpstreamThrottle429(`http://capacity-${i}.invalid/v1/chat/completions`, H_EMPTY, null, T0); + } + assert.equal(activeThrottleHolds().length, 64, "confirmed account holds fill the bounded table"); + + const model = "model-b"; + noteUpstreamThrottle429(URL_B, H_CRED_B, null, T0, model); + assert.ok(activeThrottleHolds().some((hold) => hold.host === "z.example" && hold.model === undefined), "a new model-local hold cannot evict A"); + assert.ok(throttleReject(URL_A, H_CRED_A, T0 + 1), "A remains locally blocked after a direct model-local insertion"); + + const settleB = beginProvisionalUpstreamThrottle429(URL_B, H_CRED_B, null, T0 + 1, model); + settleB(model); + assert.ok(activeThrottleHolds().some((hold) => hold.host === "z.example" && hold.model === undefined), "a settled model-local hold cannot evict A"); + assert.ok(throttleReject(URL_A, H_CRED_A, T0 + 2), "A remains locally blocked after provisional B settlement"); + assert.equal( + activeThrottleHolds().some((hold) => hold.host === "keyring-b.example" && hold.model === undefined), + false, + "capacity fallback does not widen verified B model evidence into an account hold", + ); + }); +}); + +test("#4204: per-account classification reservations cap continuations without gating unrelated traffic", () => { + withKnobs({ BILI_THROTTLE_429: "1", BILI_THROTTLE_429_MAX_CONSECUTIVE: "5", BILI_THROTTLE_429_BASE_DELAY_MS: "1000", BILI_THROTTLE_429_MAX_DELAY_MS: "8000" }, () => { + _resetUpstreamThrottleForTest(); + noteUpstreamThrottle429(URL_A, H_CRED_A, null, T0); + const model = "model-b"; + const reservations: Array<{ release(): void }> = []; + const settlers: Array<(modelScope?: string) => void> = []; + for (let i = 0; i < 64; i++) { + const reservation = reserveUpstreamThrottleClassification(URL_B, H_CRED_B); + assert.ok(reservation, "each of B's first 64 classification reservations is admitted"); + reservations.push(reservation); + settlers.push(beginProvisionalUpstreamThrottle429(URL_B, H_CRED_B, null, T0 + i, model, undefined, reservation)); + } + + assert.equal(reserveUpstreamThrottleClassification(URL_B, H_CRED_B), undefined, "the 65th active classification for B is refused"); + const aReservation = reserveUpstreamThrottleClassification(URL_A, H_CRED_A); + assert.ok(aReservation, "B's saturated per-account capacity does not consume A's classification slots"); + aReservation.release(); + assert.equal( + throttleReject("http://unrelated.example/v1/chat/completions", H_EMPTY, T0 + 10_000), + undefined, + "B's exhausted capacity does not install a process-wide gate", + ); + const pending = activeThrottleHolds(); + assert.equal(pending.length, 2, "64 reserved observations still share one B provisional gate"); + assert.ok(pending.some((hold) => hold.host === "z.example" && hold.model === undefined), "A's account hold remains recorded"); + assert.ok(throttleReject(URL_A, H_CRED_A, T0 + 100), "A remains locally blocked while B is saturated"); + + settlers[0](model); + const reopenedSlot = reserveUpstreamThrottleClassification(URL_B, H_CRED_B); + assert.ok(reopenedSlot, "settling B's first observation releases only its reservation"); + reopenedSlot.release(); + for (let i = 1; i < settlers.length; i++) settlers[i](model); + + const settled = activeThrottleHolds(); + assert.equal(settled.length, 2, "all exact results converge onto one model-local B hold"); + assert.ok(settled.some((hold) => hold.host === "z.example" && hold.model === undefined), "A is preserved after all B settlements"); + assert.ok(settled.some((hold) => hold.host === "keyring-b.example" && hold.model === model), "B's verified model is retained"); + assert.ok(throttleReject(URL_A, H_CRED_A, T0 + 100), "A remains locally blocked after B settlement"); + assert.equal(throttleReject(URL_B, H_CRED_B, T0 + 100, "other-model"), undefined, "unrelated B models remain eligible"); + for (const reservation of reservations) reservation.release(); + }); +}); + +test("#4204: reserved ambiguous settlement records account scope and releases its own slot", () => { + withKnobs({ BILI_THROTTLE_429: "1", BILI_THROTTLE_429_MAX_CONSECUTIVE: "5", BILI_THROTTLE_429_BASE_DELAY_MS: "1000", BILI_THROTTLE_429_MAX_DELAY_MS: "8000" }, () => { + _resetUpstreamThrottleForTest(); + noteUpstreamThrottle429(URL_A, H_CRED_A, null, T0); + const reservation = reserveUpstreamThrottleClassification(URL_B, H_CRED_B); + assert.ok(reservation); + const settle = beginProvisionalUpstreamThrottle429(URL_B, H_CRED_B, null, T0 + 1, "model-b", undefined, reservation); + settle(undefined); + + const holds = activeThrottleHolds(); + assert.ok(holds.some((hold) => hold.host === "z.example" && hold.model === undefined), "A's confirmed account hold remains recorded"); + assert.ok(holds.some((hold) => hold.host === "keyring-b.example" && hold.model === undefined), "ambiguous B evidence is confirmed at account scope"); + assert.ok(throttleReject(URL_A, H_CRED_A, T0 + 2), "A remains locally blocked"); + assert.ok(throttleReject(URL_B, H_CRED_B, T0 + 2, "other-model"), "B's ambiguous evidence holds every model"); + + const replacement = reserveUpstreamThrottleClassification(URL_B, H_CRED_B); + assert.ok(replacement, "settlement releases the reservation"); + replacement.release(); + }); +}); + +test("#4204: classification reservations cap active account keys at 64", () => { + withKnobs({ BILI_THROTTLE_429: "1" }, () => { + _resetUpstreamThrottleForTest(); + const reservations: Array<{ release(): void }> = []; + for (let i = 0; i < 64; i++) { + const reservation = reserveUpstreamThrottleClassification(`http://classification-${i}.invalid/v1/chat/completions`, H_EMPTY); + assert.ok(reservation, "each of the first 64 account keys is admitted"); + reservations.push(reservation); + } + + const overflowUrl = "http://classification-overflow.invalid/v1/chat/completions"; + assert.equal(reserveUpstreamThrottleClassification(overflowUrl, H_EMPTY), undefined, "a 65th active account key is refused"); + assert.equal(throttleReject(URL_B, H_CRED_B, T0 + 1), undefined, "active classification reservations do not themselves create throttle holds"); + reservations[0].release(); + const admitted = reserveUpstreamThrottleClassification(overflowUrl, H_EMPTY); + assert.ok(admitted, "releasing one account key makes room for another"); + admitted.release(); + for (const reservation of reservations.slice(1)) reservation.release(); + }); +}); + test("#4204: disabled knob restores legacy behavior — observations are no-ops, gate always allows", () => { withKnobs({ BILI_THROTTLE_429: "0" }, () => { _resetUpstreamThrottleForTest(); + const reservation = reserveUpstreamThrottleClassification(URL_A, H_EMPTY); + assert.ok(reservation, "disabled throttling returns a no-op reservation rather than capacity failure"); + reservation.release(); noteUpstreamThrottle429(URL_A, H_EMPTY, null, T0); noteUpstreamThrottle429(URL_A, H_EMPTY, null, T0); assert.equal(activeThrottleHolds().length, 0, "disabled: no state recorded"); - assert.equal(upstreamThrottleLocalReject(URL_A, H_EMPTY, T0 + 100), undefined, "disabled: never gated"); + assert.equal(throttleReject(URL_A, H_EMPTY, T0 + 100), undefined, "disabled: never gated"); }); }); @@ -194,7 +597,7 @@ test("#4204: bounded state table — 65 origins evict the oldest, no crash", () noteUpstreamThrottle429(`http://h${i}.invalid/v1/chat/completions`, H_EMPTY, null, T0 + i); } assert.equal(activeThrottleHolds().length, 64, "table bounded at 64"); - assert.equal(upstreamThrottleLocalReject("http://h0.invalid/v1/chat/completions", H_EMPTY, T0 + 1), undefined, "oldest origin evicted → no hold"); - assert.ok(upstreamThrottleLocalReject("http://h64.invalid/v1/chat/completions", H_EMPTY, T0 + 65), "newest origin still held"); + assert.equal(throttleReject("http://h0.invalid/v1/chat/completions", H_EMPTY, T0 + 1), undefined, "oldest origin evicted → no hold"); + assert.ok(throttleReject("http://h64.invalid/v1/chat/completions", H_EMPTY, T0 + 65), "newest origin still held"); }); }); diff --git a/tests/issue4725-loop-429-observe.test.ts b/tests/issue4725-loop-429-observe.test.ts index 9ee143c9a1..43dcedc7ca 100644 --- a/tests/issue4725-loop-429-observe.test.ts +++ b/tests/issue4725-loop-429-observe.test.ts @@ -3,7 +3,11 @@ import test from "node:test"; process.env.NODE_ENV = "test"; -import { noteUpstreamThrottle429, noteUpstreamThrottleResolved, upstreamThrottleLocalReject } from "../src/upstream-throttle.ts"; +import { noteUpstreamThrottle429, noteUpstreamThrottleResolved, upstreamThrottleGate } from "../src/upstream-throttle.ts"; + +function throttleReject(url: string, headers: Record) { + return upstreamThrottleGate(url, headers).reject; +} // N3 (PR #4725 round-3 review): the 429 governor's observe pair was wired only // on the relay lane; the loop lane's own sends (compression re-requests, @@ -24,13 +28,13 @@ test("429 governor contract: consecutive 429 verdicts arm the local gate; a reso const headers: Record = {}; // resolved verdict first: no hold exists — stays clean noteUpstreamThrottleResolved(url, headers, 200); - assert.equal(upstreamThrottleLocalReject(url, headers), undefined, "no hold before any 429"); + assert.equal(throttleReject(url, headers), undefined, "no hold before any 429"); // arm the hold with loop-shaped 429 verdicts (streak 2 → backoff armed) noteUpstreamThrottle429(url, headers, null); noteUpstreamThrottle429(url, headers, null); - const reject = upstreamThrottleLocalReject(url, headers); + const reject = throttleReject(url, headers); assert.ok(reject, "hold armed after consecutive 429 verdicts"); // a settled success (exactly what the loop notes on 2xx) lifts the hold noteUpstreamThrottleResolved(url, headers, 200); - assert.equal(upstreamThrottleLocalReject(url, headers), undefined, "resolved verdict lifts the hold"); + assert.equal(throttleReject(url, headers), undefined, "resolved verdict lifts the hold"); }); diff --git a/tests/issue4743-degenerate-refetch-429.test.ts b/tests/issue4743-degenerate-refetch-429.test.ts index 675d6ae304..4c8721738e 100644 --- a/tests/issue4743-degenerate-refetch-429.test.ts +++ b/tests/issue4743-degenerate-refetch-429.test.ts @@ -2,11 +2,17 @@ import assert from "node:assert/strict"; import http from "node:http"; import { once } from "node:events"; import test from "node:test"; +import { setImmediate } from "node:timers/promises"; process.env.NODE_ENV = "test"; import { makeContinuationRefetch } from "../src/degenerate-retry.ts"; -import { noteUpstreamThrottle429, noteUpstreamThrottleResolved, upstreamThrottleLocalReject } from "../src/upstream-throttle.ts"; +import { withFetchTransport } from "../src/fetch-transport.ts"; +import { _resetUpstreamThrottleForTest, activeThrottleHolds, noteUpstreamThrottle429, noteUpstreamThrottleResolved, reserveUpstreamThrottleClassification, upstreamThrottleGate, type ThrottleClassificationReservation } from "../src/upstream-throttle.ts"; + +function throttleReject(url: string, headers: Record, now?: number, requestModel?: string) { + return upstreamThrottleGate(url, headers, now, requestModel).reject; +} // #4743 (PR #4725 round-3 residue, cherry-picked from #4752 ab6a7f10): the // degenerate-turn continuation refetch is an egress send like any other, but @@ -23,13 +29,14 @@ import { noteUpstreamThrottle429, noteUpstreamThrottleResolved, upstreamThrottle const CHAT_BODY = JSON.stringify({ model: "m", messages: [{ role: "user", content: "hi" }] }); -function opts(url: string, label: string) { +function opts(url: string, label: string, signal: AbortSignal = new AbortController().signal) { return { protocol: "openai" as const, body: CHAT_BODY, + model: "m", upstreamUrl: url, reqHeaders: {}, - signal: new AbortController().signal, + signal, log: () => {}, label, }; @@ -59,7 +66,7 @@ test("#4743 A: an armed 429 hold answers the continuation refetch locally — th // loop-lane contract test: two consecutive verdicts) noteUpstreamThrottle429(srv.url, {}, null); noteUpstreamThrottle429(srv.url, {}, null); - assert.ok(upstreamThrottleLocalReject(srv.url, {}), "precondition: hold armed"); + assert.ok(throttleReject(srv.url, {}), "precondition: hold armed"); const refetch = makeContinuationRefetch(opts(srv.url, "gate-4743")); const out = await refetch(); @@ -72,27 +79,420 @@ test("#4743 A: an armed 429 hold answers the continuation refetch locally — th }); test("#4743 B: the refetch's own 429 verdict arms the governor; a resolved retry lifts the hold", async () => { - const rejected = await recordedServer(429, JSON.stringify({ error: { message: "rate limited" } })); + const rejected = await recordedServer(429, JSON.stringify({ error: { code: "keyring_all_limited", model: "m", message: "all keys limited" } })); const ok = await recordedServer(200, "event: message_start\ndata: {}\n\n"); try { - assert.equal(upstreamThrottleLocalReject(rejected.url, {}), undefined, "precondition: no hold yet"); + assert.equal(throttleReject(rejected.url, {}), undefined, "precondition: no hold yet"); const first = await makeContinuationRefetch(opts(rejected.url, "storm-4743"))(); assert.equal(first, null, "a rejected retry passes the empty turn through"); - const armed = upstreamThrottleLocalReject(rejected.url, {}); - assert.ok(armed, "the refetch's 429 verdict must reach the governor (arm for the next main-lane send)"); + const armed = throttleReject(rejected.url, {}, Date.now(), "m"); + assert.ok(armed, "keyring_all_limited remains an origin+credential hold"); + assert.ok(throttleReject(rejected.url, {}, Date.now(), "other-model"), "account quota holds all models"); const second = await makeContinuationRefetch(opts(ok.url, "lift-4743"))(); assert.ok(second, "an ok retry still yields its stream"); // the ok settle resolved the (different-host) governor key of its own // upstream; verify the observe pair by checking the rejected host's // hold is still armed until an explicit resolved verdict lifts it - assert.ok(upstreamThrottleLocalReject(rejected.url, {}), "hold on the storming host persists until a resolved verdict"); + assert.ok(throttleReject(rejected.url, {}, Date.now(), "m"), "hold on the storming host persists until a resolved verdict"); noteUpstreamThrottleResolved(rejected.url, {}, 200); - assert.equal(upstreamThrottleLocalReject(rejected.url, {}), undefined, "resolved verdict lifts the hold"); + assert.equal(throttleReject(rejected.url, {}), undefined, "resolved verdict lifts the account-wide hold"); } finally { await rejected.close(); await ok.close(); } }); + +test("#4743 C: a structured KeyRing model refusal holds only the matching model", async () => { + const rejected = await recordedServer( + 429, + JSON.stringify({ error: { code: "keyring_model_temporarily_unavailable", model: "m", message: "model unavailable" } }), + ); + try { + assert.equal(throttleReject(rejected.url, {}, Date.now(), "m"), undefined, "precondition: model has no hold"); + assert.equal(throttleReject(rejected.url, {}, Date.now(), "other-model"), undefined, "precondition: other model has no hold"); + + const first = await makeContinuationRefetch(opts(rejected.url, "model-limited-refetch"))(); + assert.equal(first, null, "a rejected continuation passes the original empty turn through"); + assert.ok(throttleReject(rejected.url, {}, Date.now(), "m"), "the matching model is held"); + assert.equal(throttleReject(rejected.url, {}, Date.now(), "other-model"), undefined, "other models remain eligible on the same credential"); + assert.equal(activeThrottleHolds()[0].model, "m", "the recorded hold retains its model scope"); + + const gated = await makeContinuationRefetch(opts(rejected.url, "model-hold-gate"))(); + assert.equal(gated, null); + assert.equal(rejected.hits(), 1, "the held model's retry is answered locally without egress"); + } finally { + await rejected.close(); + noteUpstreamThrottleResolved(rejected.url, {}, 200, Date.now(), "m"); + } +}); + +// Real timers are intentional: both concurrent response bodies finish after +// the former 1s cutoff, while the continuation callbacks stop waiting at 250ms. +test("#4743 D: concurrent delayed KeyRing bodies over 1s refine their own provisional holds without delaying callbacks", { timeout: 5_000 }, async () => { + const url = "http://delayed-keyring-429.invalid/v1/chat/completions"; + const host = new URL(url).host; + const timers: Array> = []; + const completedModels = new Set(); + const canceledModels = new Set(); + const { promise: bothRequests, resolve: resolveBothRequests } = Promise.withResolvers(); + let calls = 0; + const { promise: firstBodyComplete, resolve: resolveFirstBody } = Promise.withResolvers(); + const { promise: bothBodiesComplete, resolve: resolveBothBodies } = Promise.withResolvers(); + const transport = async () => { + const index = calls++; + if (calls === 2) resolveBothRequests(); + await bothRequests; + const model = index === 0 ? "m" : "n"; + const finishDelay = index === 0 ? 1_300 : 1_500; + return new Response( + new ReadableStream({ + start(controller) { + const encoder = new TextEncoder(); + timers.push(setTimeout(() => { + if (!canceledModels.has(model)) controller.enqueue(encoder.encode('{"error":{"code":"keyring_')); + }, 350)); + timers.push(setTimeout(() => { + if (!canceledModels.has(model)) { + controller.enqueue(encoder.encode(`model_temporarily_unavailable","model":"${model}","message":"model unavailable"}}`)); + } + }, 400)); + timers.push(setTimeout(() => { + if (!canceledModels.has(model)) { + completedModels.add(model); + controller.close(); + if (model === "m") resolveFirstBody(); + if (completedModels.size === 2) resolveBothBodies(); + } + }, finishDelay)); + }, + cancel() { + canceledModels.add(model); + }, + }), + { status: 429, headers: { "content-type": "application/json", "retry-after": "10" } }, + ); + }; + const mController = new AbortController(); + const nController = new AbortController(); + const mOpts = { ...opts(url, "delayed-m-4743"), signal: mController.signal }; + const nOpts = { ...opts(url, "delayed-n-4743"), model: "n", signal: nController.signal }; + let mProbeToken: symbol | undefined; + let nProbeToken: symbol | undefined; + + try { + const started = Date.now(); + const results = await withFetchTransport(transport, () => Promise.all([ + makeContinuationRefetch(mOpts)(), + makeContinuationRefetch(nOpts)(), + ])); + assert.deepEqual(results, [null, null]); + assert.ok(Date.now() - started < 600, "both callbacks return near the 250ms initial body budget"); + assert.equal(completedModels.size, 0, "both bodies are still incomplete when callbacks return"); + assert.ok(activeThrottleHolds().some((hold) => hold.host === host && hold.model === undefined), "pending observations retain provisional account scope"); + + await firstBodyComplete; + let holds = activeThrottleHolds().filter((hold) => hold.host === host); + for (let attempt = 0; attempt < 20 && !holds.some((hold) => hold.model === "m"); attempt++) { + await setImmediate(); + holds = activeThrottleHolds().filter((hold) => hold.host === host); + } + assert.ok(holds.some((hold) => hold.model === "m"), "the first complete exact body refines after 1s"); + assert.ok(holds.some((hold) => hold.model === undefined), "the still-pending second observation keeps its provisional account evidence"); + + await bothBodiesComplete; + holds = activeThrottleHolds().filter((hold) => hold.host === host); + for (let attempt = 0; attempt < 20 && (!holds.some((hold) => hold.model === "m") || !holds.some((hold) => hold.model === "n")); attempt++) { + await setImmediate(); + holds = activeThrottleHolds().filter((hold) => hold.host === host); + } + assert.deepEqual(holds.map((hold) => hold.model).sort(), ["m", "n"], "each complete response narrows only its own observation"); + const mAdmission = upstreamThrottleGate(url, {}, Date.now(), "m"); + mProbeToken = mAdmission.probeToken; + assert.ok(mAdmission.reject, "the matching model is held"); + const nAdmission = upstreamThrottleGate(url, {}, Date.now(), "n"); + nProbeToken = nAdmission.probeToken; + assert.ok(nAdmission.reject, "the other exact model is also held"); + assert.equal(upstreamThrottleGate(url, {}, Date.now(), "other-model").reject, undefined, "unrelated models stay eligible"); + } finally { + mController.abort(); + nController.abort(); + for (const timer of timers) clearTimeout(timer); + noteUpstreamThrottleResolved(url, {}, 200, Date.now(), "m", mProbeToken); + noteUpstreamThrottleResolved(url, {}, 200, Date.now(), "n", nProbeToken); + } +}); + +test("#4743 E: an oversized 429 body with a never-settling cancellation promise returns promptly with account scope", { timeout: 1_000 }, async () => { + const url = "http://oversized-keyring-429.invalid/v1/chat/completions"; + let cancelCalled = false; + const transport = async () => new Response( + new ReadableStream({ + start(controller) { + controller.enqueue(new Uint8Array(16 * 1024 + 1)); + }, + cancel() { + cancelCalled = true; + return new Promise(() => {}); + }, + }), + { status: 429, headers: { "content-type": "application/json" } }, + ); + + try { + const started = Date.now(); + const out = await withFetchTransport(transport, () => makeContinuationRefetch(opts(url, "oversized-scope-4743"))()); + assert.equal(out, null); + assert.ok(Date.now() - started < 500, "cancellation is fire-and-forget rather than part of callback latency"); + assert.equal(cancelCalled, true, "the oversized body is canceled"); + const hold = activeThrottleHolds().find((entry) => entry.host === new URL(url).host); + assert.ok(hold, "the ambiguous oversized response still arms a hold"); + assert.equal(hold.model, undefined, "oversized error bodies remain account-wide"); + assert.ok(throttleReject(url, {}, Date.now(), "m")); + assert.ok(throttleReject(url, {}, Date.now(), "other-model")); + } finally { + noteUpstreamThrottleResolved(url, {}, 200, Date.now(), "m"); + } +}); + +test("#4743 F: an aborted retry releases its own probe token without clearing the model hold", { timeout: 5_000 }, async () => { + const url = "http://aborted-keyring-429.invalid/v1/chat/completions"; + const previousToggle = process.env.BILI_THROTTLE_429; + process.env.BILI_THROTTLE_429 = "1"; + _resetUpstreamThrottleForTest(); + const controller = new AbortController(); + const started = Promise.withResolvers(); + const transport = async (_url: string, options: { signal?: AbortSignal | null }): Promise => { + const signal = options.signal; + assert.ok(signal, "fetch transport receives the relay abort signal"); + return await new Promise((_resolve, reject) => { + if (signal.aborted) { + started.resolve(); + reject(new Error("request aborted")); + return; + } + signal.addEventListener("abort", () => reject(new Error("request aborted")), { once: true }); + started.resolve(); + }); + }; + + try { + noteUpstreamThrottle429(url, {}, null, Date.now() - 60_000, "m"); + assert.ok(activeThrottleHolds().some((hold) => hold.model === "m"), "the model-scoped hold is armed before the probe"); + const pendingSend = withFetchTransport(transport, () => makeContinuationRefetch(opts(url, "abort-probe-4743", controller.signal))()); + await started.promise; + controller.abort(); + assert.equal(await pendingSend, null, "an aborted refetch passes the empty turn through"); + + const nextAdmission = upstreamThrottleGate(url, {}, Date.now(), "m"); + assert.equal(nextAdmission.reject, undefined, "the cancelled request releases its in-flight probe claim"); + assert.equal(typeof nextAdmission.probeToken, "symbol", "the model hold remains armed and grants a new owned probe"); + assert.ok(activeThrottleHolds().some((hold) => hold.model === "m"), "cancellation does not clear the model hold"); + } finally { + controller.abort(); + _resetUpstreamThrottleForTest(); + if (previousToggle === undefined) delete process.env.BILI_THROTTLE_429; + else process.env.BILI_THROTTLE_429 = previousToggle; + } +}); +test("#4204 continuation classifications stay isolated when one origin saturates", { timeout: 5_000 }, async () => { + const previousToggle = process.env.BILI_THROTTLE_429; + process.env.BILI_THROTTLE_429 = "1"; + _resetUpstreamThrottleForTest(); + const saturatedUrl = "http://saturated-keyring-429.invalid/v1/chat/completions"; + const otherUrl = "http://other-keyring-429.invalid/v1/chat/completions"; + const bodies: ReadableStreamDefaultController[] = []; + const allStarted = Promise.withResolvers(); + let saturatedHits = 0; + let otherHits = 0; + const attempts: Array | null>> = []; + const transport = async (url: string): Promise => { + if (url === saturatedUrl) { + saturatedHits++; + return new Response(new ReadableStream({ + start(controller) { + bodies.push(controller); + if (bodies.length === 64) allStarted.resolve(); + }, + }), { status: 429, headers: { "content-type": "application/json" } }); + } + otherHits++; + return new Response("event: message_start\ndata: {}\n\n", { status: 200 }); + }; + + try { + await withFetchTransport(transport, async () => { + for (let i = 0; i < 64; i++) { + attempts.push(makeContinuationRefetch(opts(saturatedUrl, `saturated-${i}`))()); + } + await allStarted.promise; + const pending = await Promise.all(attempts); + assert.ok(pending.every((body) => body === null), "slow 429 retries return the original empty turns promptly"); + + const saturatedRetry = await makeContinuationRefetch(opts(saturatedUrl, "saturated-overflow"))(); + assert.equal(saturatedRetry, null, "the 65th same-credential retry is declined before egress"); + assert.equal(saturatedHits, 64, "saturated same-origin admission adds no upstream request"); + + const otherBody = await makeContinuationRefetch(opts(otherUrl, "other-provider"))(); + assert.ok(otherBody, "another origin's continuation still receives its upstream stream"); + await otherBody.cancel(); + assert.equal(otherHits, 1, "the unrelated origin proceeds while saturated bodies remain unresolved"); + assert.equal(throttleReject(otherUrl, {}, Date.now(), "m"), undefined, "saturation does not create a misleading hold for the other origin"); + + const accountLimit = new TextEncoder().encode(JSON.stringify({ error: { code: "keyring_all_limited" } })); + for (const body of bodies) { + body.enqueue(accountLimit); + body.close(); + } + + const reopened: ThrottleClassificationReservation[] = []; + for (let attempt = 0; attempt < 20 && reopened.length < 64; attempt++) { + while (reopened.length < 64) { + const reservation = reserveUpstreamThrottleClassification(saturatedUrl, {}); + if (reservation === undefined) break; + reopened.push(reservation); + } + if (reopened.length < 64) await setImmediate(); + } + assert.equal(reopened.length, 64, "every completed slow body releases its reserved classification slot"); + for (const reservation of reopened) reservation.release(); + + const accountHold = activeThrottleHolds().find((hold) => hold.host === new URL(saturatedUrl).host); + assert.ok(accountHold, "ambiguous saturated 429 bodies retain account-wide evidence"); + assert.equal(accountHold.model, undefined, "account-wide quota is not narrowed to a model"); + assert.ok(throttleReject(saturatedUrl, {}, Date.now(), "other-model"), "the new account-limit evidence blocks all models on its credential"); + assert.equal(throttleReject(otherUrl, {}, Date.now(), "m"), undefined, "the saturated origin's evidence does not affect another provider"); + }); + } finally { + for (const body of bodies) { + try { + if (body.desiredSize !== null) { + body.enqueue(new TextEncoder().encode("{}")); + body.close(); + } + } catch { /* best-effort cleanup for a failed assertion */ } + } + await Promise.allSettled(attempts); + _resetUpstreamThrottleForTest(); + if (previousToggle === undefined) delete process.env.BILI_THROTTLE_429; + else process.env.BILI_THROTTLE_429 = previousToggle; + } +}); + +test("#4204 continuation classification reservations release on every exit", { timeout: 5_000 }, async () => { + const previousToggle = process.env.BILI_THROTTLE_429; + process.env.BILI_THROTTLE_429 = "1"; + _resetUpstreamThrottleForTest(); + const assertReleased = async ( + label: string, + run: (url: string) => Promise | null>, + ): Promise => { + _resetUpstreamThrottleForTest(); + const url = `http://reservation-release-${label}.invalid/v1/chat/completions`; + const seeded: ThrottleClassificationReservation[] = []; + const reopened: ThrottleClassificationReservation[] = []; + try { + for (let i = 0; i < 63; i++) { + const reservation = reserveUpstreamThrottleClassification(url, {}); + assert.ok(reservation, `${label}: seed reservation ${i} is admitted`); + seeded.push(reservation); + } + const body = await run(url); + if (body !== null) await body.cancel(); + + for (let attempt = 0; attempt < 20 && reopened.length < 1; attempt++) { + while (reopened.length < 1) { + const reservation = reserveUpstreamThrottleClassification(url, {}); + if (reservation === undefined) break; + reopened.push(reservation); + } + if (reopened.length < 1) await setImmediate(); + } + assert.equal(reopened.length, 1, `${label}: the retry's classification reservation is released on exit`); + } finally { + for (const reservation of seeded) reservation.release(); + for (const reservation of reopened) reservation.release(); + _resetUpstreamThrottleForTest(); + } + }; + + try { + await assertReleased("local-gate", async (url) => { + noteUpstreamThrottle429(url, {}, null); + let hits = 0; + const body = await withFetchTransport(async () => { + hits++; + return new Response("ok"); + }, () => makeContinuationRefetch(opts(url, "reservation-local-gate"))()); + assert.equal(hits, 0, "a locally held retry does not reach transport"); + return body; + }); + await assertReleased("resign-error", async (url) => withFetchTransport(async () => { + throw new Error("unexpected egress"); + }, () => makeContinuationRefetch({ + ...opts(url, "reservation-resign-error"), + resign: () => { throw new Error("resign failed"); }, + })())); + await assertReleased("fetch-error", async (url) => withFetchTransport(async () => { + throw new Error("fetch failed"); + }, () => makeContinuationRefetch(opts(url, "reservation-fetch-error"))())); + await assertReleased("abort", async (url) => { + const controller = new AbortController(); + const started = Promise.withResolvers(); + const transport = async (_url: string, options: { signal?: AbortSignal | null }): Promise => { + const signal = options.signal; + assert.ok(signal, "the transport receives the retry abort signal"); + const { promise, reject } = Promise.withResolvers(); + if (signal.aborted) { + started.resolve(); + reject(new Error("request aborted")); + return promise; + } + signal.addEventListener("abort", () => reject(new Error("request aborted")), { once: true }); + started.resolve(); + return promise; + }; + const pending = withFetchTransport(transport, () => makeContinuationRefetch(opts(url, "reservation-abort", controller.signal))()); + await started.promise; + controller.abort(); + return pending; + }); + await assertReleased("non-429", async (url) => withFetchTransport(async () => new Response("error", { status: 500 }), + () => makeContinuationRefetch(opts(url, "reservation-non-429"))())); + await assertReleased("success", async (url) => withFetchTransport(async () => new Response("event: message_start\ndata: {}\n\n", { status: 200 }), + () => makeContinuationRefetch(opts(url, "reservation-success"))())); + await assertReleased("settled-429", async (url) => withFetchTransport(async () => new Response("{}", { status: 429 }), + () => makeContinuationRefetch(opts(url, "reservation-settled-429"))())); + } finally { + _resetUpstreamThrottleForTest(); + if (previousToggle === undefined) delete process.env.BILI_THROTTLE_429; + else process.env.BILI_THROTTLE_429 = previousToggle; + } +}); +test("#4743 remains model-scoped when Promise.withResolvers is unavailable", async () => { + const previousToggle = process.env.BILI_THROTTLE_429; + const withResolversDescriptor = Object.getOwnPropertyDescriptor(Promise, "withResolvers"); + process.env.BILI_THROTTLE_429 = "1"; + _resetUpstreamThrottleForTest(); + const url = "http://without-promise-with-resolvers.invalid/v1/chat/completions"; + + try { + Object.defineProperty(Promise, "withResolvers", { configurable: true, writable: true, value: undefined }); + const body = JSON.stringify({ error: { code: "keyring_model_temporarily_unavailable", model: "m" } }); + const result = await withFetchTransport(async () => new Response(body, { status: 429 }), + () => makeContinuationRefetch(opts(url, "without-with-resolvers"))()); + assert.equal(result, null); + const holds = activeThrottleHolds().filter((hold) => hold.host === new URL(url).host); + assert.deepEqual(holds.map((hold) => hold.model), ["m"], "the exact model refusal is retained without the newer Promise API"); + assert.ok(throttleReject(url, {}, Date.now() - 60_000, "m"), "the refused model remains held"); + assert.equal(throttleReject(url, {}, Date.now() - 60_000, "other-model"), undefined, "unrelated models stay eligible"); + } finally { + if (withResolversDescriptor === undefined) Reflect.deleteProperty(Promise, "withResolvers"); + else Object.defineProperty(Promise, "withResolvers", withResolversDescriptor); + _resetUpstreamThrottleForTest(); + if (previousToggle === undefined) delete process.env.BILI_THROTTLE_429; + else process.env.BILI_THROTTLE_429 = previousToggle; + } +}); diff --git a/tests/proxy-noinject-tag-strip.test.ts b/tests/proxy-noinject-tag-strip.test.ts index f947c1be9d..0cfffd55d8 100644 --- a/tests/proxy-noinject-tag-strip.test.ts +++ b/tests/proxy-noinject-tag-strip.test.ts @@ -11,6 +11,7 @@ import { startServer } from "../src/server.ts"; import type { ProxyOptions } from "../src/config.ts"; import { SessionStore, _setStoreForTest } from "../src/persist.ts"; import { _setForTest as setRegistryForTest } from "../src/registry.ts"; +import { activeThrottleHolds, _resetUpstreamThrottleForTest } from "../src/upstream-throttle.ts"; // Render-tag pieces are assembled from hex escapes so no literal tag sequence // appears in this file's source. @@ -23,6 +24,7 @@ const CLOSE_MARK = `${LT}/acp${GT}`; interface Harness { proxyPort: number; upstreamPort: number; + hits(): number; close(): Promise; } @@ -34,12 +36,18 @@ function openaiChunk(data: unknown): string { return `data: ${JSON.stringify(data)}\n\n`; } -async function startHarness(compress: { injectTool: boolean; injectNudge: boolean }, scripts: string[][], json: boolean): Promise { +async function startHarness( + compress: { injectTool: boolean; injectNudge: boolean }, + scripts: string[][], + json: boolean, + statuses: number[] = [], +): Promise { let call = 0; const upstream = http.createServer((req, res) => { req.on("data", () => {}); req.on("end", () => { - res.writeHead(200, { "content-type": json ? "application/json" : "text/event-stream" }); + const status = statuses[call] ?? 200; + res.writeHead(status, { "content-type": status === 429 ? "application/json" : json ? "application/json" : "text/event-stream" }); const script = scripts[Math.min(call, scripts.length - 1)]!; call += 1; for (const line of script) res.write(line); @@ -80,6 +88,7 @@ async function startHarness(compress: { injectTool: boolean; injectNudge: boolea return { proxyPort, upstreamPort, + hits: () => call, close: async () => { proxy.close(); proxy.closeIdleConnections?.(); @@ -426,3 +435,133 @@ test("#475 G3: fake-completion buffering + streaming request answered with JSON await h.close(); } }); +test("#4204 side-lane continuation 429 remains scoped to its request model", async () => { + const previousThrottle = { + BILI_THROTTLE_429: process.env.BILI_THROTTLE_429, + BILI_THROTTLE_429_MAX_CONSECUTIVE: process.env.BILI_THROTTLE_429_MAX_CONSECUTIVE, + BILI_THROTTLE_429_BASE_DELAY_MS: process.env.BILI_THROTTLE_429_BASE_DELAY_MS, + BILI_THROTTLE_429_MAX_DELAY_MS: process.env.BILI_THROTTLE_429_MAX_DELAY_MS, + }; + process.env.BILI_THROTTLE_429 = "1"; + process.env.BILI_THROTTLE_429_MAX_CONSECUTIVE = "5"; + process.env.BILI_THROTTLE_429_BASE_DELAY_MS = "1000"; + process.env.BILI_THROTTLE_429_MAX_DELAY_MS = "8000"; + _resetUpstreamThrottleForTest(); + + const modelError = JSON.stringify({ + error: { code: "keyring_model_temporarily_unavailable", model: "model-a", message: "model unavailable" }, + }); + const modelATagOnlyTurn = [ + openaiChunk({ id: "chatcmpl_side_a", object: "chat.completion.chunk", model: "model-a", choices: [{ index: 0, delta: { role: "assistant" } }] }), + openaiChunk({ id: "chatcmpl_side_a", object: "chat.completion.chunk", model: "model-a", choices: [{ index: 0, delta: { content: TAG("m00009") } }] }), + openaiChunk({ id: "chatcmpl_side_a", object: "chat.completion.chunk", model: "model-a", choices: [{ index: 0, delta: {}, finish_reason: "stop" }] }), + "data: [DONE]\n\n", + ]; + const healthyModelB = [ + openaiChunk({ id: "chatcmpl_ok", object: "chat.completion.chunk", model: "model-b", choices: [{ index: 0, delta: { role: "assistant", content: "ok" } }] }), + openaiChunk({ id: "chatcmpl_ok", object: "chat.completion.chunk", model: "model-b", choices: [{ index: 0, delta: {}, finish_reason: "stop" }] }), + "data: [DONE]\n\n", + ]; + const h = await startHarness( + { injectTool: false, injectNudge: false }, + [modelATagOnlyTurn, [modelError], healthyModelB], + false, + [200, 429, 200], + ); + try { + const side = await fetch(`http://127.0.0.1:${h.proxyPort}/bili/http://127.0.0.1:${h.upstreamPort}/v1/chat/completions`, { + method: "POST", + headers: { "content-type": "application/json", "x-acp-session": "side-throttle-model", authorization: "Bearer same-key" }, + body: JSON.stringify({ model: "model-a", max_tokens: 100, stream: true, messages: [{ role: "user", content: "title?" }] }), + }); + assert.equal(side.status, 200); + await side.text(); + assert.equal(h.hits(), 2, "the empty side turn performs exactly one continuation refetch"); + assert.equal(activeThrottleHolds().length, 1); + assert.equal(activeThrottleHolds()[0].model, "model-a", "the continuation's exact KeyRing marker scopes the hold"); + + const otherModel = await fetch(`http://127.0.0.1:${h.proxyPort}/bili/http://127.0.0.1:${h.upstreamPort}/v1/chat/completions`, { + method: "POST", + headers: { "content-type": "application/json", "x-acp-session": "side-throttle-other", authorization: "Bearer same-key" }, + body: JSON.stringify({ model: "model-b", max_tokens: 1024, stream: true, messages: [{ role: "user", content: "answer?" }] }), + }); + assert.equal(otherModel.status, 200, "the side retry must not widen its model-local hold to the shared credential"); + await otherModel.text(); + assert.equal(h.hits(), 3, "model B reaches the upstream instead of being locally blocked"); + assert.equal(activeThrottleHolds().length, 1); + assert.equal(activeThrottleHolds()[0].model, "model-a", "model B success does not clear model A's hold"); + } finally { + _resetUpstreamThrottleForTest(); + await h.close(); + for (const [key, value] of Object.entries(previousThrottle)) { + if (value === undefined) delete process.env[key]; + else process.env[key] = value; + } + } +}); + +test("#4204 Google side-lane stream keeps its path model through continuation throttling", async () => { + const previousThrottle = { + BILI_THROTTLE_429: process.env.BILI_THROTTLE_429, + BILI_THROTTLE_429_MAX_CONSECUTIVE: process.env.BILI_THROTTLE_429_MAX_CONSECUTIVE, + BILI_THROTTLE_429_BASE_DELAY_MS: process.env.BILI_THROTTLE_429_BASE_DELAY_MS, + BILI_THROTTLE_429_MAX_DELAY_MS: process.env.BILI_THROTTLE_429_MAX_DELAY_MS, + }; + process.env.BILI_THROTTLE_429 = "1"; + process.env.BILI_THROTTLE_429_MAX_CONSECUTIVE = "5"; + process.env.BILI_THROTTLE_429_BASE_DELAY_MS = "1000"; + process.env.BILI_THROTTLE_429_MAX_DELAY_MS = "8000"; + _resetUpstreamThrottleForTest(); + const modelError = JSON.stringify({ + error: { code: "keyring_model_temporarily_unavailable", model: "path-side-a", message: "model unavailable" }, + }); + const googleTagOnlyTurn = [ + `data: ${JSON.stringify({ + modelVersion: "path-side-a", + candidates: [{ index: 0, content: { role: "model", parts: [{ text: TAG("m00009") }] }, finishReason: "STOP" }], + })}\n\n`, + ]; + const googleHealthyTurn = [ + `data: ${JSON.stringify({ + modelVersion: "path-side-b", + candidates: [{ index: 0, content: { role: "model", parts: [{ text: "ok" }] }, finishReason: "STOP" }], + })}\n\n`, + ]; + const h = await startHarness( + { injectTool: false, injectNudge: false }, + [googleTagOnlyTurn, [modelError], googleHealthyTurn], + false, + [200, 429, 200], + ); + try { + const first = await fetch(`http://127.0.0.1:${h.proxyPort}/bili/http://127.0.0.1:${h.upstreamPort}/v1beta/models/path-side-a:streamGenerateContent?alt=sse`, { + method: "POST", + headers: { "content-type": "application/json", "x-acp-session": "google-side-a", authorization: "Bearer google-side-key" }, + body: JSON.stringify({ contents: [{ role: "user", parts: [{ text: "title?" }] }], generationConfig: { maxOutputTokens: 64 } }), + }); + assert.equal(first.status, 200); + const firstText = await first.text(); + assert.equal(firstText.includes(OPEN_MARK), false, "path-streamed side response still takes the render-tag strip pipe"); + assert.equal(h.hits(), 2, "the tag-only side stream issues one continuation refetch"); + assert.equal(activeThrottleHolds().length, 1); + assert.equal(activeThrottleHolds()[0].model, "path-side-a", "the continuation 429 uses the Gemini URL model"); + + const second = await fetch(`http://127.0.0.1:${h.proxyPort}/bili/http://127.0.0.1:${h.upstreamPort}/v1beta/models/path-side-b:streamGenerateContent?alt=sse`, { + method: "POST", + headers: { "content-type": "application/json", "x-acp-session": "google-side-b", authorization: "Bearer google-side-key" }, + body: JSON.stringify({ contents: [{ role: "user", parts: [{ text: "answer?" }] }], generationConfig: { maxOutputTokens: 64 } }), + }); + assert.equal(second.status, 200, "the model-A continuation hold does not block another model on the same credential"); + await second.text(); + assert.equal(h.hits(), 3); + assert.equal(activeThrottleHolds().length, 1); + assert.equal(activeThrottleHolds()[0].model, "path-side-a"); + } finally { + _resetUpstreamThrottleForTest(); + await h.close(); + for (const [key, value] of Object.entries(previousThrottle)) { + if (value === undefined) delete process.env[key]; + else process.env[key] = value; + } + } +}); diff --git a/tests/upstream-error-log.test.ts b/tests/upstream-error-log.test.ts index 2064f2b0a4..7b0fcec282 100644 --- a/tests/upstream-error-log.test.ts +++ b/tests/upstream-error-log.test.ts @@ -8,6 +8,7 @@ import type { ProxyOptions } from "../src/config.ts"; import { SessionStore, _setStoreForTest } from "../src/persist.ts"; import { _setForTest as setRegistryForTest } from "../src/registry.ts"; import { setLogCapture } from "../src/logger.ts"; +import { activeThrottleHolds, _resetUpstreamThrottleForTest, noteUpstreamThrottle429 } from "../src/upstream-throttle.ts"; /** #2 field report: a relay answered 400 "请求参数无效" for 34 minutes and * bili.log carried zero trace, because non-2xx upstream responses were @@ -24,7 +25,11 @@ interface Captured { msg: string; } -async function startProxyWith(upstream: http.Server): Promise<{ proxy: http.Server; proxyPort: number; upstreamPort: number; captured: Captured[] }> { +async function startProxyWith( + upstream: http.Server, + models: Record = { "gpt-test": { context: 400_000 } }, + protocol?: "anthropic", +): Promise<{ proxy: http.Server; proxyPort: number; upstreamPort: number; captured: Captured[] }> { const captured: Captured[] = []; setLogCapture((level, msg) => captured.push({ level, msg })); _setStoreForTest(new SessionStore({ enabled: false })); @@ -37,7 +42,7 @@ async function startProxyWith(upstream: http.Server): Promise<{ proxy: http.Serv host: "127.0.0.1", upstream: "http://127.0.0.1", routes: { - [`http://127.0.0.1:${upstreamPort}`]: { models: { "gpt-test": { context: 400_000 } } }, + [`http://127.0.0.1:${upstreamPort}`]: { models, ...(protocol === undefined ? {} : { protocol }) }, }, modelContextLimit: 400_000, kernelConfig: defaultConfig(400_000), @@ -63,11 +68,21 @@ async function startProxyWith(upstream: http.Server): Promise<{ proxy: http.Serv return { proxy, proxyPort, upstreamPort, captured }; } -async function postChat(proxyPort: number, upstreamPort: number): Promise { +async function postChat( + proxyPort: number, + upstreamPort: number, + model = "gpt-test", + sessionId = "upstream-error-log-1", + authorization?: string, +): Promise { return fetch(`http://127.0.0.1:${proxyPort}/bili/http://127.0.0.1:${upstreamPort}/v1/chat/completions`, { method: "POST", - headers: { "content-type": "application/json", "x-acp-session": "upstream-error-log-1" }, - body: JSON.stringify({ model: "gpt-test", messages: [{ role: "user", content: "hi" }] }), + headers: { + "content-type": "application/json", + "x-acp-session": sessionId, + ...(authorization === undefined ? {} : { authorization }), + }, + body: JSON.stringify({ model, messages: [{ role: "user", content: "hi" }] }), }); } @@ -138,3 +153,497 @@ test("non-2xx upstream: oversized body passes through fully; snippet is capped a await close(upstream); } }); + +test("KeyRing model 429 throttles only the matching model; keyring_all_limited remains account-wide", async () => { + const previousThrottle = { + BILI_THROTTLE_429: process.env.BILI_THROTTLE_429, + BILI_THROTTLE_429_MAX_CONSECUTIVE: process.env.BILI_THROTTLE_429_MAX_CONSECUTIVE, + BILI_THROTTLE_429_BASE_DELAY_MS: process.env.BILI_THROTTLE_429_BASE_DELAY_MS, + BILI_THROTTLE_429_MAX_DELAY_MS: process.env.BILI_THROTTLE_429_MAX_DELAY_MS, + }; + process.env.BILI_THROTTLE_429 = "1"; + process.env.BILI_THROTTLE_429_MAX_CONSECUTIVE = "5"; + process.env.BILI_THROTTLE_429_BASE_DELAY_MS = "1000"; + process.env.BILI_THROTTLE_429_MAX_DELAY_MS = "8000"; + _resetUpstreamThrottleForTest(); + const seenModels: string[] = []; + const modelError = JSON.stringify({ + error: { code: "keyring_model_temporarily_unavailable", model: "model-a", message: "model unavailable" }, + }); + const accountError = JSON.stringify({ + error: { code: "keyring_all_limited", model: "monthly", message: "all keys are limited" }, + }); + const upstream = http.createServer((req, res) => { + const chunks: Buffer[] = []; + req.on("data", (chunk: Buffer) => chunks.push(chunk)); + req.on("end", () => { + const parsed: unknown = JSON.parse(Buffer.concat(chunks).toString("utf8")); + const model = parsed && typeof parsed === "object" && "model" in parsed && typeof parsed.model === "string" + ? parsed.model + : ""; + seenModels.push(model); + if (model === "model-a" || model === "monthly") { + res.writeHead(429, { "content-type": "application/json", "retry-after": "60" }); + res.end(model === "model-a" ? modelError : accountError); + return; + } + res.writeHead(200, { "content-type": "application/json" }); + res.end(JSON.stringify({ + id: "chatcmpl-test", + object: "chat.completion", + model, + choices: [{ index: 0, message: { role: "assistant", content: "ok" }, finish_reason: "stop" }], + })); + }); + }); + const models = { + "model-a": { context: 400_000 }, + "model-b": { context: 400_000 }, + monthly: { context: 400_000 }, + }; + let proxy: http.Server | undefined; + try { + const rig = await startProxyWith(upstream, models); + proxy = rig.proxy; + const auth = "Bearer integration-key"; + + const modelLimited = await postChat(rig.proxyPort, rig.upstreamPort, "model-a", "scope-model-a", auth); + assert.equal(modelLimited.status, 429); + assert.equal(await modelLimited.text(), modelError, "the upstream KeyRing error remains transparent"); + + const otherModel = await postChat(rig.proxyPort, rig.upstreamPort, "model-b", "scope-model-b", auth); + assert.equal(otherModel.status, 200, "a model-local refusal must not throttle another model on this credential"); + await otherModel.text(); + assert.deepEqual(seenModels, ["model-a", "model-b"]); + assert.equal(activeThrottleHolds().length, 1); + assert.equal(activeThrottleHolds()[0].model, "model-a"); + + const modelRetry = await postChat(rig.proxyPort, rig.upstreamPort, "model-a", "scope-model-a-retry", auth); + assert.equal(modelRetry.status, 429); + const localModelHold = JSON.parse(await modelRetry.text()).error; + assert.equal(localModelHold.code, "bili_throttle_hold"); + assert.equal(localModelHold.scope, "model", "local 429 includes an explicit model-scope marker for a chained relay"); + assert.equal(localModelHold.model, "model-a"); + assert.deepEqual(seenModels, ["model-a", "model-b"], "a repeat for the limited model is held without egress"); + + _resetUpstreamThrottleForTest(); + const accountLimited = await postChat(rig.proxyPort, rig.upstreamPort, "monthly", "scope-monthly", auth); + assert.equal(accountLimited.status, 429); + assert.equal(await accountLimited.text(), accountError); + const otherBlocked = await postChat(rig.proxyPort, rig.upstreamPort, "model-b", "scope-monthly-other", auth); + assert.equal(otherBlocked.status, 429, "keyring_all_limited remains a host+credential hold"); + assert.equal(JSON.parse(await otherBlocked.text()).error.code, "bili_throttle_hold"); + assert.deepEqual(seenModels, ["model-a", "model-b", "monthly"], "account quota blocks every model before egress"); + assert.equal(activeThrottleHolds()[0].model, undefined); + } finally { + _resetUpstreamThrottleForTest(); + setLogCapture(null); + if (proxy !== undefined) await close(proxy); + await close(upstream); + for (const [key, value] of Object.entries(previousThrottle)) { + if (value === undefined) delete process.env[key]; + else process.env[key] = value; + } + } +}); + +test("v1internal outer model stays model-scoped after its request envelope is unwrapped", async () => { + const previousThrottle = { + BILI_THROTTLE_429: process.env.BILI_THROTTLE_429, + BILI_THROTTLE_429_MAX_CONSECUTIVE: process.env.BILI_THROTTLE_429_MAX_CONSECUTIVE, + BILI_THROTTLE_429_BASE_DELAY_MS: process.env.BILI_THROTTLE_429_BASE_DELAY_MS, + BILI_THROTTLE_429_MAX_DELAY_MS: process.env.BILI_THROTTLE_429_MAX_DELAY_MS, + }; + process.env.BILI_THROTTLE_429 = "1"; + process.env.BILI_THROTTLE_429_MAX_CONSECUTIVE = "5"; + process.env.BILI_THROTTLE_429_BASE_DELAY_MS = "1000"; + process.env.BILI_THROTTLE_429_MAX_DELAY_MS = "8000"; + _resetUpstreamThrottleForTest(); + const seenModels: string[] = []; + const modelError = JSON.stringify({ + error: { code: "keyring_model_temporarily_unavailable", model: "model-a", message: "model unavailable" }, + }); + const upstream = http.createServer((req, res) => { + const chunks: Buffer[] = []; + req.on("data", (chunk: Buffer) => chunks.push(chunk)); + req.on("end", () => { + const parsed = JSON.parse(Buffer.concat(chunks).toString("utf8")) as { model?: unknown }; + const model = typeof parsed.model === "string" ? parsed.model : ""; + seenModels.push(model); + if (model === "model-a") { + res.writeHead(429, { "content-type": "application/json", "retry-after": "60" }); + res.end(modelError); + return; + } + res.writeHead(200, { "content-type": "application/json" }); + res.end(JSON.stringify({ + candidates: [{ index: 0, content: { role: "model", parts: [{ text: "ok" }] }, finishReason: "STOP" }], + })); + }); + }); + let proxy: http.Server | undefined; + try { + const rig = await startProxyWith(upstream, { + "model-a": { context: 400_000 }, + "model-b": { context: 400_000 }, + }); + proxy = rig.proxy; + const convId = "3f9c1a6e-4b2d-4e8a-9c1f-2d5e8b7a6c4d"; + const postV1Internal = (model: string, sessionId: string) => fetch( + `http://127.0.0.1:${rig.proxyPort}/bili/http://127.0.0.1:${rig.upstreamPort}/v1internal:generateContent`, + { + method: "POST", + headers: { + "content-type": "application/json", + "x-acp-session": sessionId, + authorization: "Bearer v1-throttle-key", + }, + body: JSON.stringify({ + project: "aicode-consumers", + requestId: `agent/${convId}`, + model, + userAgent: "antigravity", + requestType: "agent", + request: { + contents: [{ role: "user", parts: [{ text: "hi" }] }], + systemInstruction: { parts: [{ text: `\nConversation ID: ${convId}\n` }] }, + generationConfig: { maxOutputTokens: 32 }, + sessionId: 1, + }, + }), + }, + ); + + const limited = await postV1Internal("model-a", "v1-throttle-a"); + assert.equal(limited.status, 429); + await limited.text(); + const other = await postV1Internal("model-b", "v1-throttle-b"); + assert.equal(other.status, 200, "the v1internal outer model must survive unwrapping as the throttle scope"); + await other.text(); + assert.deepEqual(seenModels, ["model-a", "model-b"]); + assert.equal(activeThrottleHolds().length, 1); + assert.equal(activeThrottleHolds()[0].model, "model-a"); + } finally { + _resetUpstreamThrottleForTest(); + setLogCapture(null); + if (proxy !== undefined) await close(proxy); + await close(upstream); + for (const [key, value] of Object.entries(previousThrottle)) { + if (value === undefined) delete process.env[key]; + else process.env[key] = value; + } + } +}); + +test("Bili local model hold marker remains model-scoped when received by a chained relay", async () => { + const previousThrottle = { + BILI_THROTTLE_429: process.env.BILI_THROTTLE_429, + BILI_THROTTLE_429_MAX_CONSECUTIVE: process.env.BILI_THROTTLE_429_MAX_CONSECUTIVE, + BILI_THROTTLE_429_BASE_DELAY_MS: process.env.BILI_THROTTLE_429_BASE_DELAY_MS, + BILI_THROTTLE_429_MAX_DELAY_MS: process.env.BILI_THROTTLE_429_MAX_DELAY_MS, + }; + process.env.BILI_THROTTLE_429 = "1"; + process.env.BILI_THROTTLE_429_MAX_CONSECUTIVE = "5"; + process.env.BILI_THROTTLE_429_BASE_DELAY_MS = "1000"; + process.env.BILI_THROTTLE_429_MAX_DELAY_MS = "8000"; + _resetUpstreamThrottleForTest(); + const seenModels: string[] = []; + const upstream = http.createServer((req, res) => { + const chunks: Buffer[] = []; + req.on("data", (chunk: Buffer) => chunks.push(chunk)); + req.on("end", () => { + const parsed = JSON.parse(Buffer.concat(chunks).toString("utf8")) as { model?: unknown }; + const model = typeof parsed.model === "string" ? parsed.model : ""; + seenModels.push(model); + if (model === "model-a") { + res.writeHead(429, { "content-type": "application/json", "retry-after": "60" }); + res.end(JSON.stringify({ + error: { type: "rate_limit_error", code: "bili_throttle_hold", scope: "model", model, message: "model-a is locally held by the prior relay" }, + })); + return; + } + res.writeHead(200, { "content-type": "application/json" }); + res.end(JSON.stringify({ + id: "chatcmpl-chain", + object: "chat.completion", + model, + choices: [{ index: 0, message: { role: "assistant", content: "ok" }, finish_reason: "stop" }], + })); + }); + }); + let proxy: http.Server | undefined; + try { + const rig = await startProxyWith(upstream, { + "model-a": { context: 400_000 }, + "model-b": { context: 400_000 }, + }); + proxy = rig.proxy; + const first = await postChat(rig.proxyPort, rig.upstreamPort, "model-a", "chain-model-a", "Bearer chain-key"); + assert.equal(first.status, 429); + assert.equal(JSON.parse(await first.text()).error.code, "bili_throttle_hold"); + + const otherModel = await postChat(rig.proxyPort, rig.upstreamPort, "model-b", "chain-model-b", "Bearer chain-key"); + assert.equal(otherModel.status, 200, "a chained model-local hold must not become an account-wide hold"); + await otherModel.text(); + assert.deepEqual(seenModels, ["model-a", "model-b"]); + assert.equal(activeThrottleHolds().length, 1); + assert.equal(activeThrottleHolds()[0].model, "model-a"); + } finally { + _resetUpstreamThrottleForTest(); + setLogCapture(null); + if (proxy !== undefined) await close(proxy); + await close(upstream); + for (const [key, value] of Object.entries(previousThrottle)) { + if (value === undefined) delete process.env[key]; + else process.env[key] = value; + } + } +}); + +test("passthrough model extraction covers large JSON and model-bearing URL paths", async () => { + const previousThrottle = { + BILI_THROTTLE_429: process.env.BILI_THROTTLE_429, + BILI_THROTTLE_429_MAX_CONSECUTIVE: process.env.BILI_THROTTLE_429_MAX_CONSECUTIVE, + BILI_THROTTLE_429_BASE_DELAY_MS: process.env.BILI_THROTTLE_429_BASE_DELAY_MS, + BILI_THROTTLE_429_MAX_DELAY_MS: process.env.BILI_THROTTLE_429_MAX_DELAY_MS, + }; + process.env.BILI_THROTTLE_429 = "1"; + process.env.BILI_THROTTLE_429_MAX_CONSECUTIVE = "5"; + process.env.BILI_THROTTLE_429_BASE_DELAY_MS = "1000"; + process.env.BILI_THROTTLE_429_MAX_DELAY_MS = "8000"; + _resetUpstreamThrottleForTest(); + const seenModels: string[] = []; + const upstream = http.createServer((req, res) => { + const chunks: Buffer[] = []; + req.on("data", (chunk: Buffer) => chunks.push(chunk)); + req.on("end", () => { + const text = Buffer.concat(chunks).toString("utf8"); + const parsed = JSON.parse(text) as { model?: unknown }; + const pathModel = /\/models\/([^:?]+):/.exec(req.url ?? "")?.[1]; + const model = typeof parsed.model === "string" ? parsed.model : pathModel ?? ""; + seenModels.push(model); + if (model.endsWith("-a")) { + res.writeHead(429, { "content-type": "application/json", "retry-after": "60" }); + res.end(JSON.stringify({ + error: { code: "keyring_model_temporarily_unavailable", model, message: "model unavailable" }, + })); + return; + } + res.writeHead(200, { "content-type": "application/json" }); + res.end(JSON.stringify({ id: "ok", model })); + }); + }); + let proxy: http.Server | undefined; + try { + const rig = await startProxyWith(upstream); + proxy = rig.proxy; + const pathLimited = await fetch( + `http://127.0.0.1:${rig.proxyPort}/bili/http://127.0.0.1:${rig.upstreamPort}/v1beta/models/path-a:generateContent`, + { + method: "POST", + headers: { "content-type": "application/json", "x-bili-passthrough": "1" }, + body: JSON.stringify({ contents: [] }), + }, + ); + assert.equal(pathLimited.status, 429); + await pathLimited.text(); + const pathOther = await fetch( + `http://127.0.0.1:${rig.proxyPort}/bili/http://127.0.0.1:${rig.upstreamPort}/v1beta/models/path-b:generateContent`, + { + method: "POST", + headers: { "content-type": "application/json", "x-bili-passthrough": "1" }, + body: JSON.stringify({ contents: [] }), + }, + ); + assert.equal(pathOther.status, 200, "Gemini model path remains the throttle key on verbatim passthrough"); + await pathOther.text(); + assert.deepEqual(seenModels, ["path-a", "path-b"]); + + _resetUpstreamThrottleForTest(); + const largeLimited = await fetch( + `http://127.0.0.1:${rig.proxyPort}/bili/http://127.0.0.1:${rig.upstreamPort}/v1/other`, + { + method: "POST", + headers: { "content-type": "application/json", authorization: "Bearer large-key" }, + body: JSON.stringify({ payload: "x".repeat(20 * 1024), model: "large-a" }), + }, + ); + assert.equal(largeLimited.status, 429); + await largeLimited.text(); + const largeOther = await fetch( + `http://127.0.0.1:${rig.proxyPort}/bili/http://127.0.0.1:${rig.upstreamPort}/v1/other`, + { + method: "POST", + headers: { "content-type": "application/json", authorization: "Bearer large-key" }, + body: JSON.stringify({ payload: "x".repeat(20 * 1024), model: "large-b" }), + }, + ); + assert.equal(largeOther.status, 200, "a >16KiB passthrough body retains its request model scope"); + await largeOther.text(); + assert.deepEqual(seenModels, ["path-a", "path-b", "large-a", "large-b"]); + assert.equal(activeThrottleHolds().length, 1); + assert.equal(activeThrottleHolds()[0].model, "large-a"); + } finally { + _resetUpstreamThrottleForTest(); + setLogCapture(null); + if (proxy !== undefined) await close(proxy); + await close(upstream); + for (const [key, value] of Object.entries(previousThrottle)) { + if (value === undefined) delete process.env[key]; + else process.env[key] = value; + } + } +}); +test("Anthropic-on-Vertex passthrough keeps the raw model-path throttle scope", async () => { + const previousThrottle = { + BILI_THROTTLE_429: process.env.BILI_THROTTLE_429, + BILI_THROTTLE_429_MAX_CONSECUTIVE: process.env.BILI_THROTTLE_429_MAX_CONSECUTIVE, + BILI_THROTTLE_429_BASE_DELAY_MS: process.env.BILI_THROTTLE_429_BASE_DELAY_MS, + BILI_THROTTLE_429_MAX_DELAY_MS: process.env.BILI_THROTTLE_429_MAX_DELAY_MS, + }; + process.env.BILI_THROTTLE_429 = "1"; + process.env.BILI_THROTTLE_429_MAX_CONSECUTIVE = "5"; + process.env.BILI_THROTTLE_429_BASE_DELAY_MS = "1000"; + process.env.BILI_THROTTLE_429_MAX_DELAY_MS = "8000"; + _resetUpstreamThrottleForTest(); + const limitedModel = "claude-path@2025-01-01"; + const otherModel = "claude-path-b@2025-01-01"; + const seenModels: string[] = []; + const upstream = http.createServer((req, res) => { + const encodedModel = /\/publishers\/anthropic\/models\/([^/:?]+):/.exec(req.url ?? "")?.[1] ?? ""; + const model = decodeURIComponent(encodedModel); + seenModels.push(model); + if (model === limitedModel) { + res.writeHead(429, { "content-type": "application/json", "retry-after": "60" }); + res.end(JSON.stringify({ + error: { code: "keyring_model_temporarily_unavailable", model, message: "model unavailable" }, + })); + return; + } + res.writeHead(200, { "content-type": "application/json" }); + res.end(JSON.stringify({ type: "message", model, content: [], stop_reason: "end_turn" })); + }); + let proxy: http.Server | undefined; + try { + const rig = await startProxyWith(upstream, { "gpt-test": { context: 400_000 } }, "anthropic"); + proxy = rig.proxy; + const first = await fetch( + `http://127.0.0.1:${rig.proxyPort}/bili/http://127.0.0.1:${rig.upstreamPort}/v1/projects/p/locations/us/publishers/anthropic/models/${encodeURIComponent(limitedModel)}:rawPredict`, + { + method: "POST", + headers: { "content-type": "application/json", "x-bili-passthrough": "1", authorization: "Bearer vertex-key" }, + body: JSON.stringify({ anthropic_version: "2023-06-01", max_tokens: 64, messages: [{ role: "user", content: "hi" }] }), + }, + ); + assert.equal(first.status, 429); + await first.text(); + const second = await fetch( + `http://127.0.0.1:${rig.proxyPort}/bili/http://127.0.0.1:${rig.upstreamPort}/v1/projects/p/locations/us/publishers/anthropic/models/${encodeURIComponent(otherModel)}:rawPredict`, + { + method: "POST", + headers: { "content-type": "application/json", "x-bili-passthrough": "1", authorization: "Bearer vertex-key" }, + body: JSON.stringify({ anthropic_version: "2023-06-01", max_tokens: 64, messages: [{ role: "user", content: "hi" }] }), + }, + ); + assert.equal(second.status, 200, "a different Vertex path model remains eligible"); + await second.text(); + assert.deepEqual(seenModels, [limitedModel, otherModel]); + assert.equal(activeThrottleHolds().length, 1); + assert.equal(activeThrottleHolds()[0].model, limitedModel); + } finally { + _resetUpstreamThrottleForTest(); + setLogCapture(null); + if (proxy !== undefined) await close(proxy); + await close(upstream); + for (const [key, value] of Object.entries(previousThrottle)) { + if (value === undefined) delete process.env[key]; + else process.env[key] = value; + } + } +}); + +test("an aborted relay probe releases only its owned token while preserving the model hold", { timeout: 5_000 }, async () => { + const previousThrottle = { + BILI_THROTTLE_429: process.env.BILI_THROTTLE_429, + BILI_THROTTLE_429_MAX_CONSECUTIVE: process.env.BILI_THROTTLE_429_MAX_CONSECUTIVE, + BILI_THROTTLE_429_BASE_DELAY_MS: process.env.BILI_THROTTLE_429_BASE_DELAY_MS, + BILI_THROTTLE_429_MAX_DELAY_MS: process.env.BILI_THROTTLE_429_MAX_DELAY_MS, + }; + process.env.BILI_THROTTLE_429 = "1"; + process.env.BILI_THROTTLE_429_MAX_CONSECUTIVE = "5"; + process.env.BILI_THROTTLE_429_BASE_DELAY_MS = "1000"; + process.env.BILI_THROTTLE_429_MAX_DELAY_MS = "8000"; + _resetUpstreamThrottleForTest(); + const firstStarted = Promise.withResolvers(); + const firstResponseClosed = Promise.withResolvers(); + let upstreamHits = 0; + const upstream = http.createServer((req, res) => { + upstreamHits++; + const first = (req.url ?? "").includes("probe=first"); + req.on("end", () => { + if (first) { + res.once("close", () => firstResponseClosed.resolve()); + firstStarted.resolve(); + return; + } + res.writeHead(200, { "content-type": "application/json" }); + res.end(JSON.stringify({ choices: [{ index: 0, message: { role: "assistant", content: "ok" }, finish_reason: "stop" }] })); + }); + req.resume(); + }); + let proxy: http.Server | undefined; + const controller = new AbortController(); + let firstRequest: Promise | undefined; + try { + const rig = await startProxyWith(upstream, { "cancel-model": { context: 400_000 } }); + proxy = rig.proxy; + const upstreamUrl = `http://127.0.0.1:${rig.upstreamPort}/v1/chat/completions`; + const authorization = "Bearer cancel-probe-key"; + noteUpstreamThrottle429(upstreamUrl, { authorization }, null, Date.now() - 60_000, "cancel-model"); + firstRequest = fetch( + `http://127.0.0.1:${rig.proxyPort}/bili/http://127.0.0.1:${rig.upstreamPort}/v1/chat/completions?probe=first`, + { + method: "POST", + headers: { "content-type": "application/json", "x-acp-session": "cancelled-probe-first", authorization }, + body: JSON.stringify({ model: "cancel-model", messages: [{ role: "user", content: "hi" }] }), + signal: controller.signal, + }, + ); + await firstStarted.promise; + controller.abort(); + await assert.rejects(firstRequest, (error: unknown) => error instanceof Error, "client cancellation aborts its relay request"); + await firstResponseClosed.promise; + await new Promise((resolve) => setImmediate(resolve)); + assert.ok(activeThrottleHolds().some((hold) => hold.model === "cancel-model"), "cancelling a probe preserves its model hold"); + + const second = await fetch( + `http://127.0.0.1:${rig.proxyPort}/bili/http://127.0.0.1:${rig.upstreamPort}/v1/chat/completions?probe=second`, + { + method: "POST", + headers: { "content-type": "application/json", "x-acp-session": "cancelled-probe-second", authorization }, + body: JSON.stringify({ model: "cancel-model", messages: [{ role: "user", content: "hi" }] }), + }, + ); + assert.equal(second.status, 200, "the next same-model request can claim a new probe after cancellation"); + await second.text(); + assert.equal(upstreamHits, 2, "the cancelled probe and subsequent admitted request each reach upstream once"); + assert.equal(activeThrottleHolds().length, 0, "the successful owner clears the matching model hold"); + } finally { + controller.abort(); + if (firstRequest !== undefined) await firstRequest.catch(() => undefined); + _resetUpstreamThrottleForTest(); + setLogCapture(null); + if (proxy !== undefined) { + proxy.closeAllConnections(); + await close(proxy); + } + if (upstream.listening) { + upstream.closeAllConnections(); + await close(upstream); + } + for (const [key, value] of Object.entries(previousThrottle)) { + if (value === undefined) delete process.env[key]; + else process.env[key] = value; + } + } +}); From 5e826f1a3863e4421307a41f75ea23496c4205b3 Mon Sep 17 00:00:00 2001 From: ci-check Date: Thu, 5 Nov 2026 05:39:01 +0800 Subject: [PATCH 2/3] fix: keep ThrottleLocalReject and UpstreamThrottleGate module-internal (#5696) Both interfaces are only referenced inside src/upstream-throttle.ts (makeReject return type, gate factory return type); nothing imports them from another module, so the unused-exports gate flags them as INTERNAL_ONLY. Drop the leading export per the gate's own instruction. --- src/upstream-throttle.ts | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/upstream-throttle.ts b/src/upstream-throttle.ts index 1afcff5542..bd54dc2b14 100644 --- a/src/upstream-throttle.ts +++ b/src/upstream-throttle.ts @@ -292,7 +292,7 @@ const NOOP_CLASSIFICATION_RESERVATION: ThrottleClassificationReservation = { release(): void {}, }; -export interface ThrottleLocalReject { +interface ThrottleLocalReject { /** 429 while under budget (Retry-After attached); 424 terminal. */ status: 429 | 424; code: "bili_throttle_hold" | "bili_throttle_terminal"; @@ -305,7 +305,7 @@ export interface ThrottleLocalReject { model?: string; } -export interface UpstreamThrottleGate { +interface UpstreamThrottleGate { reject?: ThrottleLocalReject; /** Present only when this send claimed one or more expired hold slots. */ probeToken?: symbol; From 1d59391eb743aa6525b416640ea9c13e7afc590c Mon Sep 17 00:00:00 2001 From: ci-check Date: Thu, 5 Nov 2026 05:39:01 +0800 Subject: [PATCH 3/3] docs: sync 429-governor config seeds with the regenerated reference (#5696) The PR updated CONFIGURATION.md/CONFIGURATION.zh-CN.md for the new model-scoped hold semantics but left the website/config-reference seeds on the old wording, so docs-sync regeneration reverted the two rows (throttle429Enabled, throttle429MaxConsecutive). Point the seeds at the committed text; generator output is now byte-exact. --- website/config-reference/process.yaml | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/website/config-reference/process.yaml b/website/config-reference/process.yaml index c8f0f653ee..58efbc38bb 100644 --- a/website/config-reference/process.yaml +++ b/website/config-reference/process.yaml @@ -87,15 +87,15 @@ entries: default: "true" env: BILI_THROTTLE_429 group: process - desc_en: "Egress-side 429 governor (#4204): after consecutive upstream 429s, further sends to that origin+credential are answered locally with zero egress instead of being forwarded; set false to restore the legacy always-forward behavior." - desc_zh: "出站 429 治理器(#4204):同一 origin+凭据连续收到上游 429 后,后续请求在本地应答(零出站)而不再转发;设为 false 恢复旧的总是转发行为。" + desc_en: "Egress-side 429 governor (#4204): repeated upstream 429s are answered locally instead of forwarded. Account limits and unrecognized/malformed responses hold origin+credential; only KeyRing's exact `keyring_model_temporarily_unavailable` code with a matching request model holds origin+credential+model, leaving other models eligible. A mismatch or missing model remains account-wide; false restores the legacy always-forward behavior." + desc_zh: "出站 429 治理器(#4204):连续上游 429 会在本地应答而不再转发。账号限额及无法识别/格式错误的响应按 origin+凭据持有;只有 KeyRing 精确返回 `keyring_model_temporarily_unavailable` 且响应模型与请求模型完全相同时,才按 origin+凭据+模型持有,让其他模型继续可用。模型不匹配或缺失仍按账号级处理;设为 false 恢复旧的总是转发行为。" - key: network.throttle429MaxConsecutive type: number default: "5" env: BILI_THROTTLE_429_MAX_CONSECUTIVE group: process - desc_en: "Consecutive upstream 429s before the hold turns terminal (local 424 outside every known SDK retry set); one probe per backoff window keeps checking until any non-429 response clears it." - desc_zh: "连续上游 429 达到该次数后 hold 转为终局(本地 424,不在任何已知 SDK 重试集合内);每个退避窗口仍放行一次探测,直到出现非 429 响应清除。" + desc_en: "Consecutive upstream 429s before the hold turns terminal (local 424 outside every known SDK retry set); one probe per backoff window keeps checking until any non-429 response clears an account hold, or the matching model's non-429 response clears its model hold." + desc_zh: "连续上游 429 达到该次数后 hold 转为终局(本地 424,不在任何已知 SDK 重试集合内);每个退避窗口仍放行一次探测,直到账号级 hold 收到任意非 429 响应,或模型级 hold 收到同一模型的非 429 响应后清除。" - key: network.throttle429BaseDelayMs type: number default: "2000"