Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ Versions follow the merge of a `*_release-v*` branch; CI publishes to npm on tag
- **The full-resend watermark fastpath (#5403/#5418) is now ON by default (GA)** after its §7.4 human review: plugin-lane chat-completions turns whose prefix is byte-stable skip the full history re-parse/re-pin on hit (the #5403 OOM producer), any doubt still falls back to the untouched full-parse path per turn. Opt out per install with `rollout.features.pluginFullResendWatermark: false` (or a percent) or `BILI_ROLLOUT_PLUGIN_FULL_RESEND_WATERMARK=0`.

### Fixed
- **KeyRing model-local 429s no longer throttle other models on the same credential.** Bili gives an independent origin+credential+model hold only to HTTP 429 responses whose structured KeyRing or Bili marker has an exact request-model match; `keyring_all_limited`, direct OpenCode monthly quotas, malformed bodies, and mismatches remain origin+credential holds. Large passthrough bodies use a top-level model scan, while Gemini/Vertex/v1 and side-lane paths carry their actual wire model. Account/model probe slots are request-owned and canceled sends release only their token; concurrent provisional 429 observations settle independently. Continuation retries reserve bounded classification capacity per origin+credential before egress; saturation skips only the optional refetch, without a global gate or discarding account-wide evidence. Continuation reads retain the upstream idle/abort policy. Side-lane SSE retains its stream mode so render-tag stripping still runs. Regressions: `tests/issue4204-throttle-governor.test.ts`, `tests/issue4743-degenerate-refetch-429.test.ts`, `tests/proxy-noinject-tag-strip.test.ts`, and `tests/upstream-error-log.test.ts`.

- **External-summary mode no longer reports protected or already-folded ranges as failed compression.** An unchanged kernel preview now returns the normal idempotent no-op or kernel-error receipt without a paid summarizer call; validated internal probe summaries prevent an irrelevant submitted-hint length from blocking that receipt. Probe-created blocks with no recoverable content still refuse rather than folding an unreviewed hint. Regression: `tests/external-summary-runtime.test.ts` (24/24).
* **#5412: 异步存储后端的「压缩冻结会话」现在也能自愈。** #5408 事故(一个会话连续 17 次摘要漂移后被保护机制冻住、8 小时无人知晓)的主修复 #5409 只教了同步存储路径(SQLite 直写)自救:漂移挂起期间保存会自动升级为全量重导入并清掉索引里的残留行。异步路径(opt-in 网络后端,如 mysql)那份孪生代码没搬——同样被冻住的会话在那上面永远救不回来,每次保存都重写不完整视图、下轮采样继续漂移。本 PR 在 `src/storage/shadow.ts` 的异步入队路径补上同一行对齐(`completeLog = hasRawTail || driftedSessions.has(id)`),行为与同步路径逐字一致;回归测试同时钉住正门(latch 挂起的折叠保存清掉外来行)与负门(未挂起保存不清扫)。
Expand Down
4 changes: 2 additions & 2 deletions CONFIGURATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -166,8 +166,8 @@ This index is generated from `website/config-reference/*.yaml` — edit the seed
| `network.preflightDeadEndCooldownMs` | number | 300000 | BILI_PREFLIGHT_DEAD_END_COOLDOWN_MS | Cooldown after a dead-end preflight before retrying reconciliation for the same session. |
| `network.replayRetryMax` | number | 3 (1 disables replays) | BILI_REPLAY_RETRY_MAX | Maximum transport-failure replays of an identical request before surfacing the error. |
| `network.replayRetryBaseMs` | number | 1500 (0 disables the delay) | BILI_REPLAY_RETRY_BASE_MS | Base backoff delay between replays (exponential). |
| `network.throttle429Enabled` | boolean | true | BILI_THROTTLE_429 | Egress-side 429 governor (#4204): after consecutive upstream 429s, further sends to that origin+credential are answered locally with zero egress instead of being forwarded; set false to restore the legacy always-forward behavior. |
| `network.throttle429MaxConsecutive` | number | 5 | BILI_THROTTLE_429_MAX_CONSECUTIVE | Consecutive upstream 429s before the hold turns terminal (local 424 outside every known SDK retry set); one probe per backoff window keeps checking until any non-429 response clears it. |
| `network.throttle429Enabled` | boolean | true | BILI_THROTTLE_429 | Egress-side 429 governor (#4204): repeated upstream 429s are answered locally instead of forwarded. Account limits and unrecognized/malformed responses hold origin+credential; only KeyRing's exact `keyring_model_temporarily_unavailable` code with a matching request model holds origin+credential+model, leaving other models eligible. A mismatch or missing model remains account-wide; false restores the legacy always-forward behavior. |
| `network.throttle429MaxConsecutive` | number | 5 | BILI_THROTTLE_429_MAX_CONSECUTIVE | Consecutive upstream 429s before the hold turns terminal (local 424 outside every known SDK retry set); one probe per backoff window keeps checking until any non-429 response clears an account hold, or the matching model's non-429 response clears its model hold. |
| `network.throttle429BaseDelayMs` | number | 2000 | BILI_THROTTLE_429_BASE_DELAY_MS | Base backoff for the 429 hold (doubles per consecutive 429, jittered into [0.5×, 1×], capped by throttle429MaxDelayMs; an upstream Retry-After wins when larger). |
| `network.throttle429MaxDelayMs` | number | 30000 | BILI_THROTTLE_429_MAX_DELAY_MS | Ceiling for the 429 hold backoff and for the terminal-phase probe interval. |
| `network.maxShrinkPerCompress` | number | unset | BILI_MAX_SHRINK_PER_COMPRESS | Cap on how much one compression may shrink history (anti-collapse guard). |
Expand Down
4 changes: 2 additions & 2 deletions CONFIGURATION.zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -166,8 +166,8 @@
| `network.preflightDeadEndCooldownMs` | number | 300000 | BILI_PREFLIGHT_DEAD_END_COOLDOWN_MS | 死胡同 preflight 后,对同一会话重试校准的冷却时长。 |
| `network.replayRetryMax` | number | 3 (1 disables replays) | BILI_REPLAY_RETRY_MAX | 同一请求在向上游暴露错误前的最大传输失败重放次数。 |
| `network.replayRetryBaseMs` | number | 1500 (0 disables the delay) | BILI_REPLAY_RETRY_BASE_MS | 重放之间的基础退避延迟(指数增长)。 |
| `network.throttle429Enabled` | boolean | true | BILI_THROTTLE_429 | 出站 429 治理器(#4204):同一 origin+凭据连续收到上游 429 后,后续请求在本地应答(零出站)而不再转发;设为 false 恢复旧的总是转发行为。 |
| `network.throttle429MaxConsecutive` | number | 5 | BILI_THROTTLE_429_MAX_CONSECUTIVE | 连续上游 429 达到该次数后 hold 转为终局(本地 424,不在任何已知 SDK 重试集合内);每个退避窗口仍放行一次探测,直到出现非 429 响应清除。 |
| `network.throttle429Enabled` | boolean | true | BILI_THROTTLE_429 | 出站 429 治理器(#4204):连续上游 429 会在本地应答而不再转发。账号限额及无法识别/格式错误的响应按 origin+凭据持有;只有 KeyRing 精确返回 `keyring_model_temporarily_unavailable` 且响应模型与请求模型完全相同时,才按 origin+凭据+模型持有,让其他模型继续可用。模型不匹配或缺失仍按账号级处理;设为 false 恢复旧的总是转发行为。 |
| `network.throttle429MaxConsecutive` | number | 5 | BILI_THROTTLE_429_MAX_CONSECUTIVE | 连续上游 429 达到该次数后 hold 转为终局(本地 424,不在任何已知 SDK 重试集合内);每个退避窗口仍放行一次探测,直到账号级 hold 收到任意非 429 响应,或模型级 hold 收到同一模型的非 429 响应后清除。 |
| `network.throttle429BaseDelayMs` | number | 2000 | BILI_THROTTLE_429_BASE_DELAY_MS | 429 hold 的基础退避(每连续一次 429 翻倍、抖动到 [0.5×, 1×]、以 throttle429MaxDelayMs 封顶;上游 Retry-After 更大时优先)。 |
| `network.throttle429MaxDelayMs` | number | 30000 | BILI_THROTTLE_429_MAX_DELAY_MS | 429 hold 退避的上限,也是终局阶段探测间隔的上限。 |
| `network.maxShrinkPerCompress` | number | unset | BILI_MAX_SHRINK_PER_COMPRESS | 单次压缩允许收缩历史的上限(防塌缩护栏)。 |
Expand Down
11 changes: 5 additions & 6 deletions src/acp-status.ts
Original file line number Diff line number Diff line change
Expand Up @@ -399,17 +399,16 @@ export function handleAcpStatus(args: Record<string, unknown>, ctx: AcpStatusCtx
}
if (openCircuits.length > 4) extra.push(`UPSTREAM CIRCUIT OPEN — (+${openCircuits.length - 4} more host(s))`);
}
// #4204: display counterpart of the egress-side 429 governor — holds are
// per origin+credential and answer locally (zero egress) while armed, so
// hosts/models can see WHY sends are held back and when the next probe
// fires. Display-only; no wire decision lives here.
// #4204: display counterpart of the egress-side 429 governor. Account
// holds are per origin+credential; a verified KeyRing model refusal has
// its own origin+credential+model hold. Both answer locally (zero egress).
const throttleHolds = activeThrottleHolds();
if (throttleHolds.length > 0) {
for (const h of throttleHolds.slice(0, 4)) {
extra.push("");
extra.push(`UPSTREAM 429 HOLD — ${h.host}: ${h.streak} consecutive upstream 429(s); outbound sends answered locally until ${new Date(h.nextAttemptAt).toISOString()} (${h.terminal ? "TERMINAL — local 424 outside SDK retry sets until a probe succeeds" : "backoff window"}); zero egress while held, clears on the next non-429 response (#4204).`);
extra.push(`UPSTREAM 429 HOLD — ${h.host}${h.model === undefined ? "" : ` (model ${h.model})`}: ${h.streak} consecutive upstream 429(s); outbound sends answered locally until ${new Date(h.nextAttemptAt).toISOString()} (${h.terminal ? "TERMINAL — local 424 outside SDK retry sets until a probe succeeds" : "backoff window"}); zero egress while held, clears on a resolved non-429 response in the same scope (#4204).`);
}
if (throttleHolds.length > 4) extra.push(`UPSTREAM 429 HOLD — (+${throttleHolds.length - 4} more origin+credential key(s))`);
if (throttleHolds.length > 4) extra.push(`UPSTREAM 429 HOLD — (+${throttleHolds.length - 4} more origin+credential[+model] key(s))`);
}
const shrinkAt = ctx.session.stats.emergencyShrinkAt;
if (typeof shrinkAt === "number") {
Expand Down
135 changes: 123 additions & 12 deletions src/degenerate-retry.ts
Original file line number Diff line number Diff line change
@@ -1,10 +1,29 @@
import { fetchWithTimeout } from "./fetch-util.js";
import { noteUpstreamThrottle429, noteUpstreamThrottleResolved, upstreamThrottleLocalReject } from "./upstream-throttle.js";
import { beginProvisionalUpstreamThrottle429, keyringModel429Scope, noteUpstreamThrottleResolved, releaseUpstreamThrottleProbe, requestModelFromBody, reserveUpstreamThrottleClassification, upstreamThrottleGate, type ThrottleClassificationReservation } from "./upstream-throttle.js";
import { hasCompactionTrigger } from "./codex-compact.js";
import { proxyDispatcher } from "./upstream-proxy.js";
import type { WireProtocol } from "./util.js";
import { appendTrailingUserText } from "./wire-body.js";

const MAX_THROTTLE_ERROR_BODY_BYTES = 16 * 1024;
// Keep callback latency separate from the bounded body read used for scope refinement.
const MAX_THROTTLE_ERROR_READ_MS = 250;

function cancelThrottleErrorReader(reader: ReadableStreamDefaultReader<Uint8Array>): void {
try {
void reader.cancel().catch(() => undefined);
} catch {
/* cancellation is best-effort; never make the client wait for it */
}
}

function releaseThrottleErrorReader(reader: ReadableStreamDefaultReader<Uint8Array>): void {
try {
reader.releaseLock();
} catch {
/* an in-flight read still owns the lock */
}
}
// #732/#821 ran this retry for the compress loop, which owns the request body
// AND the response framing. The plugin pipe (plugin mode, and proxy-mode chat
// SSE that skipped compress injection) owns neither: it forwards the agent's
Expand Down Expand Up @@ -34,6 +53,8 @@ interface ContinuationRetryOpts {
protocol: WireProtocol;
/** The body that was forwarded upstream (the agent's own, in plugin mode). */
body: string | Buffer;
/** Exact request model; required to isolate a matching KeyRing model refusal. */
model?: string;
upstreamUrl: string;
reqHeaders: Record<string, string>;
proxyUrl?: string;
Expand All @@ -51,6 +72,36 @@ interface ContinuationRetryOpts {
wrapBody?: (body: string | Buffer) => string | Buffer;
}

async function readSmallThrottleErrorBody(response: Response): Promise<Uint8Array | undefined> {
const reader = response.body?.getReader();
if (!reader) return undefined;
const chunks: Uint8Array[] = [];
let bytes = 0;
try {
while (true) {
const result = await reader.read();
if (result.done) {
const body = new Uint8Array(bytes);
let offset = 0;
for (const chunk of chunks) {
body.set(chunk, offset);
offset += chunk.byteLength;
}
return body;
}
bytes += result.value.byteLength;
if (bytes > MAX_THROTTLE_ERROR_BODY_BYTES) {
cancelThrottleErrorReader(reader);
return undefined;
}
chunks.push(result.value);
}
} catch {
return undefined;
} finally {
releaseThrottleErrorReader(reader);
}
}
/** Build the re-request used when a turn ends with no visible output. Returns a
* function for the stream pipe: it yields the retry's response body, or null
* when the retry cannot be issued (unbuildable body, upstream error, client
Expand Down Expand Up @@ -81,16 +132,31 @@ export function makeContinuationRefetch(opts: ContinuationRetryOpts): () => Prom
return null;
}
const wireRetryBody = opts.wrapBody ? opts.wrapBody(retryBody) : retryBody;
const requestModel = opts.model ?? requestModelFromBody(retryBody);
let probeToken: symbol | undefined;
let classificationReservation: ThrottleClassificationReservation | undefined;
let settleProvisional: ((modelScope?: string) => void) | undefined;
try {
classificationReservation = reserveUpstreamThrottleClassification(opts.upstreamUrl, opts.reqHeaders);
if (classificationReservation === undefined) {
opts.log("info", `[${opts.label}] [plugin] degenerate-terminal retry skipped: throttle classification capacity is full`);
return null;
}
opts.resign?.(opts.reqHeaders, wireRetryBody);
// #4743: the retry is an egress send like any other — feed the 429
// governor (observe only, no local gate: the retry is bounded by
// the degenerate-turn ladder, and holding IT locally would strand
// the client's already-open response stream). A 429 verdict arms
// the hold for the NEXT main-lane send; any other resolved status
// clears it — same one-observation-per-request discipline as relay.
const throttleReject = upstreamThrottleLocalReject(opts.upstreamUrl, opts.reqHeaders);
// #4743: the retry is an egress send like any other. Its local
// admission gate returns a token so only this send can release the
// probe slots it claimed.
const throttleGate = upstreamThrottleGate(
opts.upstreamUrl,
opts.reqHeaders,
Date.now(),
requestModel,
);
probeToken = throttleGate.probeToken;
const throttleReject = throttleGate.reject;
if (throttleReject !== undefined) {
classificationReservation.release();
classificationReservation = undefined;
opts.log("info", `[${opts.label}] [plugin] degenerate-terminal retry answered locally by the 429 hold (#4204) — passing the empty turn through`);
return null;
}
Expand All @@ -106,13 +172,55 @@ export function makeContinuationRefetch(opts: ContinuationRetryOpts): () => Prom
opts.signal,
);
if (!r.response.ok || !r.response.body) {
r.clearTimer();
if (r.response.status === 429) noteUpstreamThrottle429(opts.upstreamUrl, opts.reqHeaders, r.response.headers.get("retry-after"));
else noteUpstreamThrottleResolved(opts.upstreamUrl, opts.reqHeaders, r.response.status);
if (r.response.status === 429) {
const retryAfter = r.response.headers.get("retry-after");
const now = Date.now();
const settle = beginProvisionalUpstreamThrottle429(
opts.upstreamUrl,
opts.reqHeaders,
retryAfter,
now,
requestModel,
probeToken,
classificationReservation,
);
classificationReservation = undefined;
settleProvisional = settle;
// Keep the fetch watchdog and external-abort listener alive
// until this bounded reader completes. Its only local wait
// is the continuation callback's 250ms latency budget.
const bodyRead = readSmallThrottleErrorBody(r.response)
.then((errorBody) => {
const modelScope = keyringModel429Scope(r.response.status, requestModel, errorBody);
settle(modelScope);
})
.catch(() => {
settle();
})
.finally(() => {
r.clearTimer();
})
.catch(() => undefined);
let initialTimer: ReturnType<typeof setTimeout> | undefined;
const firstDeadline = new Promise<{ kind: "timeout" }>((resolve) => {
initialTimer = setTimeout(() => resolve({ kind: "timeout" }), MAX_THROTTLE_ERROR_READ_MS);
});
await Promise.race([
bodyRead.then(() => ({ kind: "body" as const })),
firstDeadline,
]).finally(() => clearTimeout(initialTimer));
} else {
noteUpstreamThrottleResolved(opts.upstreamUrl, opts.reqHeaders, r.response.status, Date.now(), requestModel, probeToken);
classificationReservation.release();
classificationReservation = undefined;
r.clearTimer();
}
opts.log("warn", `[${opts.label}] [plugin] degenerate-terminal retry rejected (HTTP ${r.response.status}); passing the empty turn through`);
return null;
}
noteUpstreamThrottleResolved(opts.upstreamUrl, opts.reqHeaders, r.response.status);
noteUpstreamThrottleResolved(opts.upstreamUrl, opts.reqHeaders, r.response.status, Date.now(), requestModel, probeToken);
classificationReservation.release();
classificationReservation = undefined;
const body = r.response.body as ReadableStream<Uint8Array>;
// The retry stream is spliced into a response the caller already
// owns, so its own idle timer is dropped — a stalled retry then ends
Expand All @@ -121,6 +229,9 @@ export function makeContinuationRefetch(opts: ContinuationRetryOpts): () => Prom
r.stopIdleTimer();
return body;
} catch (e) {
settleProvisional?.();
classificationReservation?.release();
releaseUpstreamThrottleProbe(probeToken);
opts.log("warn", `[${opts.label}] [plugin] degenerate-terminal retry failed (${e instanceof Error ? e.message : String(e)}); passing the empty turn through`);
return null;
}
Expand Down
Loading
Loading