Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CONFIGURATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -1259,6 +1259,7 @@ For each request, the proxy resolves the settings by longest-URL-prefix match (t
- **Default:** *(the model's native window)*
- **Status:** ACTIVE
- **Description:** The context window size, in tokens. This is the **denominator** the engine uses for its usage ratio (`usage = tokens / modelContextLimit`) — it is **not** a truncation cap. Accepts an absolute number (`200000`) or a percent string (`"80%"` = 80% of the model's native window, resolved from the built-in table or models.dev registry). When omitted at every level, the native window is used. This is the highest-priority source for the model limit; it overrides the built-in table, the legacy per-model `context` field, and the top-level `modelContextLimit`. Note it also serves as the **hard preflight wall**: once a payload reaches this value the proxy proactively folds context before forwarding, and if folding cannot bring it under, the request fails fast instead of being sent upstream. To keep day-to-day context small while still letting large reads burst up to the native window, keep `modelContextLimit` at its native value and use `maxContextLimit` as the soft target instead (see [Soft target with elastic headroom](#soft-target-with-elastic-headroom-1122)).
- **Overflow-arm behavior (#3450).** When upstream rejects a payload with a stated-window error, the proxy arms that rejection for the retry: if the window declared here differs from the one upstream states, it logs a one-time warning naming both values and folds against the smaller of the two; if the error body also states token counts (e.g. `you requested N tokens (M in the messages, K in the completion)`), `M` is booked as the measured payload size (a ceiling is not a size) and `K` is subtracted from this wall for fit judgments while the arm lives — upstream proved it bills the completion share against the same budget. The next real usage report retires the arm and the reservation together. If your declaration is wrong, set this key to the true window: the warning goes away and the fold target follows.

#### `outputHeadroomMaxPct`

Expand Down
1 change: 1 addition & 0 deletions CONFIGURATION.zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -1262,6 +1262,7 @@ ACP 原生 agent(当前为 `pi` 扩展)会在每个进程内向代理上报
- **默认值:** *(模型的原始窗口)*
- **状态:** ACTIVE
- **说明:** 上下文窗口大小,以 token 为单位。它是引擎用于计算使用率比例的**分母**(`usage = tokens / modelContextLimit`)—— 它**不是**截断上限。接受绝对数值(`200000`)或百分比字符串(`"80%"` = 模型原始窗口的 80%,从内置表或 models.dev 注册表解析)。在每个层级都省略时,使用原始窗口。这是模型上限的最高优先级来源;它会覆盖内置表、旧版按模型的 `context` 字段以及顶层的 `modelContextLimit`。注意它同时也是**预检压缩的硬墙**:一旦载荷达到该值,代理会在转发前主动折叠上下文;若折叠后仍超出,请求会直接快速失败而不是发往上游。若希望日常上下文保持较小、同时允许大读取任务突发到原始窗口,请让 `modelContextLimit` 保持为原始窗口值,改用 `maxContextLimit` 作为软目标(见[软目标与弹性余量](#软目标与弹性余量-1122))。
- **溢出武装行为 (#3450)。** 当上游以"窗口超限"错误拒绝载荷时,代理会把这次拒绝"武装"给下一次尝试:若此处声明的窗口与上游给出的数值不一致,会记一条一次性警告(同时给出两个值)并按两者中较小者折叠;若错误正文还给出了 token 计数(如 `you requested N tokens (M in the messages, K in the completion)`),则把 `M` 记为实测载荷大小(上限不是尺寸),并在该武装存续期间从本墙中扣除 `K` 再判断载荷是否装得下——上游已证明完成份额与输入共用同一预算。下一次真实的 usage 报告会同时退役该武装与预留。如果你的声明有误,把本键设为真实窗口即可:警告消失,折叠目标随之修正。

#### `outputHeadroomMaxPct`

Expand Down
5 changes: 5 additions & 0 deletions src/cache-ledger.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1524,7 +1524,12 @@ export function settleUsageReport(
// anchor immediately — no need to wait out its prepare budget.
clearPostRebuildAnchor(session);
// #1110: a real usage report retires the one-shot overflow arm.
// #3450: the proven completion reservation retires with it — it was
// evidence about the rejected payload's turn; a recovered session
// returns to judging on the declared window (a repeat overflow
// re-arms both in one event).
delete session.stats.overflowArmTokens;
delete session.stats.overflowArmCompletionReserve;
}
// #1933 F1: roll the pending pair forward to the current turn's value —
// it will be consumed by the NEXT report (or cleared when the lane
Expand Down
5 changes: 5 additions & 0 deletions src/persist.ts
Original file line number Diff line number Diff line change
Expand Up @@ -143,6 +143,8 @@ interface PersistedSession {
lastInputTokensSource?: string;
lastInputTokensOrigin?: string;
overflowArmTokens?: number;
overflowArmCompletionReserve?: number;
overflowArmMismatchWarned?: boolean;
contextTokens?: number;
contextTokensSource?: string;
// #2129: measurement-state fields buildRecord persists via spread —
Expand Down Expand Up @@ -2488,6 +2490,9 @@ function buildSession(parsed: PersistedSession): Session {
lastInputTokensSource: stats.lastInputTokensSource === "usage" || stats.lastInputTokensSource === "estimate" || stats.lastInputTokensSource === "overflow-arm" ? stats.lastInputTokensSource : undefined,
// #1110: one-shot overflow arm — legacy files lack it; absent = no arm.
overflowArmTokens: typeof stats.overflowArmTokens === "number" && Number.isFinite(stats.overflowArmTokens) && stats.overflowArmTokens > 0 ? stats.overflowArmTokens : undefined,
// #3450: proven completion reservation + one-time warn flag — legacy files lack both.
overflowArmCompletionReserve: typeof stats.overflowArmCompletionReserve === "number" && Number.isFinite(stats.overflowArmCompletionReserve) && stats.overflowArmCompletionReserve > 0 ? stats.overflowArmCompletionReserve : undefined,
overflowArmMismatchWarned: stats.overflowArmMismatchWarned === true ? true : undefined,
// In-memory only — a fresh process has no pending compress fold.
compressCreditTokens: 0,
contextTokens: Math.max(0, stats.contextTokens ?? parsed.contextTokens ?? 0),
Expand Down
5 changes: 5 additions & 0 deletions src/preflight.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2071,6 +2071,11 @@ export async function preflightCompress(deps: PreflightDeps, messages: CoreMessa
const compressed = deps.session.stats.compressCreditTokens - creditBefore;
const folded = baselineKnown ? compressed : messages.filter((message) => effectiveMessageIds.includes(message.id)).reduce((total, message) => total + (message.text ?? "").length, 0);
currentTokens = Math.max(0, currentTokens - folded + countText(summary));
// The apply path (stream.ts applyRanges) already netted the gross
// fold out of lastInputTokens and credited compressCreditTokens;
// add the re-entering summary's cost back so the baseline tracks
// the post-fold payload size exactly once (#3450 trace: arm 1048576
// → −gross +summary = 1047168 after one fold).
deps.session.stats.lastInputTokens += defaultCountTokens(summary);
appliedThisRound += 1;
result.compressedRanges += 1;
Expand Down
22 changes: 20 additions & 2 deletions src/server.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2995,7 +2995,25 @@ export async function preflightCompressIfNeeded(
instanceId: string,
): Promise<Prepared | PreflightFailFast> {
const session = prepared.session;
const limit = config.modelContextLimit;
// #3450: when an upstream-stated overflow proved this provider bills the
// COMPLETION against the same wall ("N in the messages, N in the
// completion"), the foldable quantity is MESSAGES — every payload-vs-wall
// judgment below runs on window − reservation instead of the raw window.
// The reservation applies MIN-wise to the stated wall, never subtracted
// from config.modelContextLimit: handle.ts rewrites that value IN PLACE
// with the output-headroom reservation (its [headroom] line: "nudge/
// preflight judge input against ${reserved}"), so subtracting the
// demonstrated completion share again would count it twice and collapse
// the wall toward zero whenever a stated count lands on a reserved
// session (#3450 × #3738 merge regression; case 3 of
// tests/fix-arm-headroom-clamp.test.ts pins the recovery). In the rare
// stated-below-declared corner the min() over-folds slightly — the safe
// direction (availability over context retention).
const statedWall = typeof session.stats.overflowArmTokens === "number" && session.stats.overflowArmTokens > 0 ? session.stats.overflowArmTokens : 0;
const completionReserve = typeof session.stats.overflowArmCompletionReserve === "number" && session.stats.overflowArmCompletionReserve > 0 ? session.stats.overflowArmCompletionReserve : 0;
const limit = completionReserve > 0 && statedWall > 0
? Math.max(0, Math.min(config.modelContextLimit, statedWall - completionReserve))
: config.modelContextLimit;
const overflowTarget = prepared.protocol === "responses" && isCodexClient(req.headers) && codexCompactMode() === "intercept"
? limit * CODEX_COMPACT_HEALTH_RATIO
: limit;
Expand Down Expand Up @@ -3314,7 +3332,7 @@ export async function preflightCompressIfNeeded(
// provider-billed baseline and the (calibrated) local estimate — so a
// false trigger is diagnosable from the log alone instead of requiring a
// cross-reference between gate and nudge lines.
log("warn", `[${session.id}] context ${tokenCount} tokens reached preflight target ${compressionTarget}${growthArmed ? " (auto-fold growth floor)" : ""} (model window ${limit}, model=${model}; usage-baseline=${baselineFloor > 0 ? baselineFloor : "none"} local-est=${Math.round(calibratedText)}${kFactor !== undefined ? ` raw=${Math.round(textEstimate + overheadEstimate)} k̂=${kFactor.toFixed(2)}` : ""}); preflight compressing before forward`);
log("warn", `[${session.id}] context ${tokenCount} tokens reached preflight target ${compressionTarget}${growthArmed ? " (auto-fold growth floor)" : ""}${completionReserve > 0 ? ` (window ${config.modelContextLimit} minus proven completion reservation ${completionReserve})` : ""} (model window ${limit}, model=${model}; usage-baseline=${baselineFloor > 0 ? baselineFloor : "none"} local-est=${Math.round(calibratedText)}${kFactor !== undefined ? ` raw=${Math.round(textEstimate + overheadEstimate)} k̂=${kFactor.toFixed(2)}` : ""}); preflight compressing before forward`);
// #300: stamp the chain marker so a downstream bili skips these
// summarization calls too (preflight always processes).
const { upstreamUrl, headers, proxyUrl } = buildForwardTarget(req, opts, route, affinity, instanceId);
Expand Down
16 changes: 15 additions & 1 deletion src/server/handle.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2359,6 +2359,17 @@ async function handleRequest(
logRequestCost(log, session.id, inboundMsgs, inboundBytes, reqT0, forwardBody);
return { body: forwardBody, prepared: null };
}
// #3450: ONE wall per request — on the refold path the
// prepare/nudge stage must judge against the same window the
// preflight stage folds on. min(declared, stated): the
// operator/host declaration and the upstream's own proof bound
// each other; neither may outrank the other within one
// request (the two-walls split let "usage=105%" and
// "target 1048576" coexist in the same second). Downward only
// — a stated ceiling above the declaration never loosens it.
if (overflowWindow !== undefined && typeof reqConfig.modelContextLimit === "number" && reqConfig.modelContextLimit > 0 && overflowWindow < reqConfig.modelContextLimit) {
reqConfig = { ...reqConfig, modelContextLimit: overflowWindow };
}
prepared = await runPrepare();
// [#2926] anthropic Threads lane: prepare mirrored the turn and
// may have armed a reforge (mirror estimate crossed the window
Expand Down Expand Up @@ -2440,7 +2451,10 @@ async function handleRequest(
res,
opts,
core,
overflowWindow !== undefined ? { ...reqConfig, modelContextLimit: overflowWindow } : reqConfig,
// #3450: no per-stage override anymore — the wall was
// unified above (min of declared and upstream-stated),
// so prepare/nudge and preflight judge on one number.
reqConfig,
nativeWindow,
requestModel,
resolvedNativeWindow,
Expand Down
38 changes: 33 additions & 5 deletions src/server/relay.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1028,21 +1028,49 @@ export async function forward(
// re-sent the same oversized history every turn — 400 forever.
// Clamping to min(stated, reserved) keeps the arm upstream-truth
// when the two agree, and only bites when a reservation exists.
//
// #3450: when the body states the REJECTED PAYLOAD'S OWN size
// ("N in the messages"), book THAT as the armed baseline — the
// window is a ceiling, not a measurement; arming at the ceiling
// made preflight see "payload ~window, deficit 0" while the real
// payload sat above its local estimate and the re-forward died.
// The #3738 clamp then caps that measurement at the judging limit
// (a stated count exceeds the reserved limit exactly when the
// completion share pushed the total over the window).
const measured = typeof info.requestedMessages === "number" && info.requestedMessages > 0 ? info.requestedMessages : undefined;
const judgingLimit = typeof s.metadata.effectiveContextLimit === "number" && s.metadata.effectiveContextLimit > 0
? s.metadata.effectiveContextLimit
: 0;
const armedWindow = judgingLimit > 0 && judgingLimit < info.window ? judgingLimit : info.window;
s.stats.lastInputTokens = armedWindow;
const unclampedBaseline = measured ?? info.window;
const armedBaseline = judgingLimit > 0 && judgingLimit < unclampedBaseline ? judgingLimit : unclampedBaseline;
s.stats.lastInputTokens = armedBaseline;
s.stats.lastInputTokensSource = "overflow-arm";
// #2313: the rejection came from THIS upstream — stamp the
// evidence origin so the #1933 F2 route gate (and the #2313
// probe-forward gate) treat the arm as current-route evidence
// instead of demoting it against a stale settle origin.
s.stats.lastInputTokensOrigin = normalizeUpstreamOrigin(upstreamUrl);
// #1110: record the arm SEPARATELY so the side-request guard
// can read it without ever touching the nudge baseline.
s.stats.overflowArmTokens = armedWindow;
log("warn", `[${s.id}] upstream context overflow (model=${reqModel ?? "unknown"}) — window ${info.window} stated upstream; armed emergency shrink at ${armedWindow}${armedWindow === info.window ? "" : ` (clamped to the output-headroom-reserved judging limit ${judgingLimit})`}, declared window unchanged (#987)`);
// can read it without ever touching the nudge baseline. The guard
// keeps the CEILING even when the baseline books a measured or
// clamped value (#3450) — it must never clamp below what the
// upstream proved, and it reserves each side request's own output
// from this value itself, so feeding it a pre-reserved number
// would double-subtract the headroom.
s.stats.overflowArmTokens = info.window;
// #3450: the completion share proves this provider bills OUTPUT
// against the same wall — remember it so preflight folds against
// window − completion (the foldable quantity is MESSAGES).
const reserve = typeof info.requestedCompletion === "number" && info.requestedCompletion > 0 ? info.requestedCompletion : undefined;
if (reserve !== undefined) s.stats.overflowArmCompletionReserve = reserve;
// #3450: one-time mismatch warning — the upstream-stated window
// differs from the window bili actually judges on.
const declaredWindow = typeof s.metadata.effectiveContextLimit === "number" ? s.metadata.effectiveContextLimit : 0;
if (declaredWindow > 0 && declaredWindow !== info.window && !s.stats.overflowArmMismatchWarned) {
s.stats.overflowArmMismatchWarned = true;
log("warn", `[${s.id}] upstream-stated window ${info.window} differs from the window in use (${declaredWindow}, model=${reqModel ?? "unknown"}) — folding targets min of the two; set compress.modelContextLimit if the declaration is wrong (#3450)`);
}
log("warn", `[${s.id}] upstream context overflow (model=${reqModel ?? "unknown"}) — window ${info.window} stated upstream${measured !== undefined ? `, rejected payload ~${measured} tokens stated` : ""}${reserve !== undefined ? `, completion reservation ${reserve} tokens` : ""}; armed emergency shrink at ${armedBaseline}${armedBaseline < unclampedBaseline ? ` (clamped to the judging limit ${judgingLimit}, #3738)` : ""}, declared window unchanged (#987)`);
} else {
// No window number stated — nothing to learn (and #987
// removed the learner anyway), but the rejection itself is
Expand Down
15 changes: 15 additions & 0 deletions src/session.ts
Original file line number Diff line number Diff line change
Expand Up @@ -335,6 +335,16 @@ export type Session = {
* usage report (one-shot, per the #554 contract) and on compression
* reset. Absent on legacy session files → treated as no arm. */
overflowArmTokens?: number;
/** #3450: completion-channel reservation PROVEN by an upstream-stated
* overflow ("N in the completion" billed against the same wall) —
* preflight folds against window − reserve because the foldable
* quantity is MESSAGES. Set alongside the overflow arm; cleared on
* native-compaction reset like the arm itself. Absent/zero → no
* reservation (legacy behavior). */
overflowArmCompletionReserve?: number;
/** #3450: one-time mismatch warning already emitted for this session
* (upstream-stated window ≠ window in use). Provenance only. */
overflowArmMismatchWarned?: boolean;
/** Tokens compressed THIS turn whose fold has not yet materialized in
* an upstream usage report (the post-compress re-request re-sends the
* UNFOLDED history for prefix-cache reasons, so its usage report
Expand Down Expand Up @@ -1247,6 +1257,11 @@ export function resetSessionCompression(session: Session): void {
// longer bounds this session; drop it so the guard falls back to the
// declared window instead of clamping to a stale ceiling.
delete session.stats.overflowArmTokens;
// #3450: same boundary — the proven completion reservation described the
// pre-compaction payload lineage, and the rebuilt conversation re-arms the
// one-time mismatch warning.
delete session.stats.overflowArmCompletionReserve;
delete session.stats.overflowArmMismatchWarned;
// #728: the pre-compaction outbound payload is gone — the old estimate
// (measured against the pre-compaction wire) would read high and blind
// the nudge fallback early; let the next prepare* re-measure.
Expand Down
Loading
Loading