Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 51 additions & 0 deletions src/preflight-verdict.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
// #3977 goal B: the preflight exit state machine. Every exit of
// preflightCompressIfNeeded is exactly one of these verdicts, decided by the
// triple 【evidence adjudication (src/size-evidence.ts), walk result
// (PreflightResult), fit measurement (fits)】 — not by if-chain position.
// Failure titles render from the FAIL_FAST verdict at a single site.
//
// Migration plan for this enum (issue #3977): it LANDS here in step 1 as the
// typed contract; step 2 routes the eight escape hatches through the
// transition table that consumes these kinds; step 3 collapses the exit sites
// onto one render. Until step 2 the preflight function still returns its
// Prepared | PreflightFailFast shapes directly — behavior is unchanged.

/** Why a payload is forwarded ONCE for upstream arbitration instead of being
* folded or refused. Each reason is one escape hatch family:
* - probe-evidence: #2313 estimate-only probe (baseline carries no current-
* route upstream evidence; forward once to acquire usage/overflow evidence)
* - image-billing: #496/#1800 image arbitration — images alone clear the
* window on estimate with no overflow evidence (immediate + post-fold gates)
* - divergence: #1839 the walk's own post-fold view fits but the rebuilt
* normal-config payload measures over
* - fresh-size: #3146 fresh (unknown-baseline) session still over-window by
* local estimate after maximum folding */
export type ForwardOnceReason = "probe-evidence" | "image-billing" | "divergence" | "fresh-size";

/** What a fail-fast verdict is FOR. The class drives retry semantics and the
* rendered title; status/retryable stay explicit fields because the same
* class can carry different statuses across paths (deterministic-exhaustion
* is always 413, transient is 502 today but is defined by cause, not code). */
export type FailFastClass =
| "aborted" // client disconnected mid-preflight (status 0, no response)
| "doomed-bytes" // #2490/#3146 doom math: no tokenizer could fit it (413, retryable=false)
| "deterministic-exhaustion" // #3461: math proved no combination of folds closes the gap (413)
| "upstream-throttle" // upstream 429 surfaced through preflight (503)
| "transient" // relay-dependent exhaustion: empty/truncated summaries, budget draws, pool saturation (502)
| "dead-end-cooldown"; // #726 cached diagnosis replayed (mechanism removed in #4079 — no exit site on current master)

export type PreflightVerdict =
/** Payload fits the window (or never triggered); forward as-is. */
| { kind: "forward-fits" }
/** Folding committed and the rebuilt payload fits; forward the rebuild. */
| { kind: "fold" }
/** Forward exactly once; the upstream arbitrates size/billing (#496 house pattern). */
| { kind: "forward-once-arbitrate"; reason: ForwardOnceReason }
/** Refuse locally with a diagnostic; nothing was forwarded. */
| { kind: "fail-fast"; status: number; retryable: boolean; cls: FailFastClass }
/** Dead-end cooldown active: identical request already proved un-foldable (#726; mechanism removed in #4079 — no exit site on current master, kind retained for ledger mapping until step 2). */
| { kind: "cooldown" }
/** Growth-armed zero progress with no foldable candidates: lean steady state, no backoff (#2638). */
| { kind: "steady-state" }
/** Growth-armed zero progress caused purely by queue drops: pool congestion, retry next turn (#2662). */
| { kind: "congestion" };
96 changes: 11 additions & 85 deletions src/server.ts
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,7 @@ export { isChatGptCodexUpstream, isCodexResponsesLite, resolvePromptCacheKey, sh
import { reconcileFoldCoverage, noteSystemPromptFingerprint, resolveFoldReconcileMode } from "./fold-reconcile.js";
import { biliToolsDeclaredOnWire, countBiliToolUses, evaluateSelfHealRound, nudgeSuppressed, pluginLaneDegraded, pluginLaneRestore } from "./session-self-heal.js";
import { getSession, hasProcessedState, listSessions, peekSession, type PendingRetrieval, type Session, initSessions, markDirty, flushAllSessions, acquireInFlight, releaseInFlight, totalInFlight, snapshotMessages, ensureCanonicalId, storeEffectiveConfig, foldCoverage, postRebuildAnchorTokens, setPostRebuildAnchor, tickPostRebuildAnchor, splitSessionWarnings, RESPONSES_TURN_SEPARATOR_ID_PREFIX } from "./session.js";
import { baselineFloor as projectBaselineFloor, hasOverflowEvidence, localRefusalCovers, noteLocalRefusal, readBaselineEvidence } from "./size-evidence.js";
import { detectStaleInstall } from "./update.js";
import { decideEaddrinuse, eaddrinuseExitCode, postBindCanLadder, UPGRADE_REPLACEMENT_ENV, type FailFastReason } from "./listen-policy.js";
import { getAdvisoryState, cannotResolveTarget } from "./advisory.js";
Expand Down Expand Up @@ -2729,79 +2730,6 @@ function preflightHoldGraceMs(): number {
// (#4079) the #726 dead-end COOLDOWN was removed entirely — its knob and
// config key with it.

// #3846: a local 413 ("can never fit, NOT forwarded") is independent evidence
// that this origin/model/window cannot accept the payload — but it only ever
// lived in the #726 dead-end marker's body-hash dimension: the moment the
// baseline source demotes to estimate (the #2313 probe re-arms at
// baselineFloor === 0) or the body changes between turns (the hash misses),
// the refusal evaporates and every forward-once escape hatch (estimate-only
// probe / image arbitration / #1839 rebuilt-view divergence / #3146
// fresh-session) re-bets the same estimator guess — production shape: three
// consecutive full-bill forwards of 606K/620K/633K-token payloads bili itself
// had already refused locally. Persist refusals as their own metadata channel
// — NEVER as usage (that field is upstream billing evidence only) — scoped to
// the route/model/window that saw them; an equal-or-wider refused window
// covers the current equal-or-tighter one, anything else evaluates fresh.
interface LocalRefusalRecord {
origin: string;
model: string;
window: number;
at: number;
}

const LOCAL_REFUSAL_MAX_RECORDS = 16;

/** (#3974/#4079) a local refusal is route-scoped CIRCUMSTANTIAL evidence, not
* a permanent property of the model: windows get re-declared (config fix,
* catalog refresh, #3993-style fallback recovery), payloads shrink, and an
* unexpiring record locked new sessions out of every forward-once escape
* hatch forever. Ten minutes covers the aggressive host-retry window the
* record exists for (#3846's consecutive full-bill re-sends) while
* guaranteeing the lock self-heals without any user action. */
const LOCAL_REFUSAL_TTL_MS = 10 * 60_000;

function readLocalRefusals(metadata: Record<string, unknown>): LocalRefusalRecord[] {
const raw = metadata.localRefusals;
if (!Array.isArray(raw)) return [];
const now = Date.now();
return raw.filter((r): r is LocalRefusalRecord =>
!!r && typeof r === "object"
&& typeof (r as LocalRefusalRecord).origin === "string"
&& typeof (r as LocalRefusalRecord).model === "string"
&& typeof (r as LocalRefusalRecord).window === "number"
// (#4079/#3974) a record without a readable timestamp is treated as
// expired too — sessions persisted before the TTL carried no `at`, and
// those are exactly the ones stuck in the never-expiring lock.
&& typeof (r as LocalRefusalRecord).at === "number"
&& (r as LocalRefusalRecord).at > now - LOCAL_REFUSAL_TTL_MS,
);
}

/** Record a local 413 refusal for (origin, model, window). A wider refusal in
* the same scope already dominates (it covers every tighter window); a
* narrower one replaces it so the record tracks the widest verdict seen. */
function noteLocalRefusal(session: Session, origin: string | undefined, model: string | undefined, window: number): void {
if (!origin || !model || !Number.isFinite(window) || window <= 0) return;
const records = readLocalRefusals(session.metadata);
const existing = records.find((r) => r.origin === origin && r.model === model);
if (existing !== undefined && existing.window >= window) return;
session.metadata.localRefusals = [
...records.filter((r) => !(r.origin === origin && r.model === model)),
{ origin, model, window: Math.floor(window), at: Date.now() },
].slice(-LOCAL_REFUSAL_MAX_RECORDS);
markDirty(session);
}

/** True when this session was locally refused (413) by the SAME origin+model
* at an equal or WIDER window — the current request may then not reopen any
* forward-once escape hatch (#3846). Both sides must be known
* (evidence-permitlist discipline: an unprovenanced refusal proves nothing
* about an unprovenanced request, and vice versa). */
function localRefusalCovers(metadata: Record<string, unknown>, origin: string | undefined, model: string | undefined, window: number): boolean {
if (!origin || !model) return false;
return readLocalRefusals(metadata).some((r) => r.origin === origin && r.model === model && r.window >= window);
}

/** #568: commit the response early so a long preflight cannot lose the client
* to its header timeout. Streaming clients get an SSE stream with keep-alive
* comment lines (`: bili-preflight` — a spec-mandated no-op for every SSE
Expand Down Expand Up @@ -3224,9 +3152,8 @@ export async function preflightCompressIfNeeded(
// (#604) on a measured (folded) payload describes a different view and
// must not pull preflight into multi-minute runs over a payload whose own
// post-fold estimate fits the window.
const baselineFloorRaw = prepared.processedMessages.length > 0
? ((session.stats.lastInputTokensSource === "usage" || session.stats.lastInputTokensSource === "overflow-arm") ? session.stats.lastInputTokens : 0)
: session.stats.lastInputTokens;
const baselineEvidence = readBaselineEvidence(session.stats);
const currentOrigin = normalizeUpstreamOrigin(route?.upstream);
// #1933 F2: a usage baseline is only authoritative for the route that
// measured it — provider billing scales differ per upstream (the incident:
// ~257K local estimate vs 59-63% real usage on one route; after a mid-
Expand All @@ -3235,11 +3162,11 @@ export async function preflightCompressIfNeeded(
// untrusted when the request now routes elsewhere; the payload's own
// (calibrated) estimate then judges it. Unprovenanced baselines (sessions
// started before this field existed) keep the legacy behavior.
let baselineFloor = baselineFloorRaw;
const currentOrigin = normalizeUpstreamOrigin(route?.upstream);
const baselineOrigin = normalizeUpstreamOrigin(session.stats.lastInputTokensOrigin);
if (baselineFloor > 0 && currentOrigin !== undefined && baselineOrigin !== undefined && baselineOrigin !== currentOrigin) {
log("info", `[${session.id}] preflight usage-baseline ~${baselineFloor} tok was measured on ${maskUrlForLog(baselineOrigin)}, request now routes to ${maskUrlForLog(currentOrigin)} — demoting to untrusted, judging by this payload's own estimate (#1933)`);
const baselineProjection = projectBaselineFloor(baselineEvidence, currentOrigin, prepared.processedMessages.length > 0);
let baselineFloor = baselineProjection.floor;
if (baselineProjection.demoted && currentOrigin !== undefined) {
// demoted ⇒ both origins are known (size-evidence.projectBaselineFloor contract)
log("info", `[${session.id}] preflight usage-baseline ~${baselineProjection.from} tok was measured on ${maskUrlForLog(baselineProjection.origin ?? "")}, request now routes to ${maskUrlForLog(currentOrigin)} — demoting to untrusted, judging by this payload's own estimate (#1933)`);
baselineFloor = 0;
}
// #1933 F1: scale the local text estimate by the per-route calibration
Expand Down Expand Up @@ -3313,7 +3240,7 @@ export async function preflightCompressIfNeeded(
// [#autoFold] a growth trigger is operator intent, not a billing guess —
// it never needs upstream evidence and must fold on the FIRST armed
// request, so the #2313 estimate-only forward-once never skips it.
const probeForEvidence = !growthArmed && !unknownBaseline && session.stats.lastInputTokens > 0 && baselineFloor <= 0 && prepared.processedMessages.length > 0;
const probeForEvidence = !growthArmed && !unknownBaseline && baselineEvidence.tokens > 0 && baselineFloor <= 0 && prepared.processedMessages.length > 0;
if (probeForEvidence) {
// #2490: mathematically-doomed probe guard. The forward-once below bets
// one request on "the estimator might be wrong about size"; that bet has
Expand Down Expand Up @@ -3379,8 +3306,7 @@ export async function preflightCompressIfNeeded(
// estimates back into lastInputTokens, which permanently closed this
// hatch on pixel-billing upstreams). With evidence present we trust the
// estimate and fall through to fold / fail-fast below.
const noOverflowEvidence = session.stats.lastInputTokens < limit
|| (session.stats.lastInputTokensSource !== "usage" && session.stats.lastInputTokensSource !== "overflow-arm");
const noOverflowEvidence = !hasOverflowEvidence(baselineEvidence, limit);
// #1800/#1843: the TEXT channel fits the window but `text + imageReserve`
// does not, and we hold no overflow evidence → the image reserve clears the
// window on ESTIMATE alone while the real bill may be far smaller (pixel-tile
Expand Down Expand Up @@ -3440,7 +3366,7 @@ export async function preflightCompressIfNeeded(
// estimator diverges from upstream billing (#3146 read chars-as-tokens
// ~2x high). Rejections backed by real evidence keep today's wording.
const estimateOnly = baselineFloor === 0 && kFactor === undefined
&& session.stats.lastInputTokensSource !== "usage" && session.stats.lastInputTokensSource !== "overflow-arm";
&& !(baselineEvidence.grade === "usage" || baselineEvidence.grade === "overflow");
const estimateNote = estimateOnly ? " (size judged by bili's local estimate, which can diverge from what the upstream actually bills)" : "";
const message =
`${sizeClause} (model=${model})${rangesClause}${estimateNote} ` +
Expand Down
Loading
Loading