diff --git a/docs-site/src/content/docs/guides/combos.md b/docs-site/src/content/docs/guides/combos.md index 6bc5f3b6ee..332dbc8c72 100644 --- a/docs-site/src/content/docs/guides/combos.md +++ b/docs-site/src/content/docs/guides/combos.md @@ -177,6 +177,7 @@ Combo failures are divided into **hop** failures and **terminal** failures. | Result | Behavior | | --- | --- | | HTTP 401, 403, 404, 408, 429, or any 5xx | Cool the target and hop to the next eligible target. | +| HTTP 410 with an explicit model end-of-life, retired, deprecated, sunset, decommissioned, or no-longer-available signal | Cool that target and hop. Unrelated 410 responses remain terminal. | | Classified authentication, subscription, quota, rate-limit, overload, or upstream-server error | Cool the target and hop, even when the status alone is not sufficient. | | Client cancellation (499), `origin_rejected`, cyber-policy refusal, context overflow, or invalid request | Stop and return the error; another target would not make the request valid. | | Any other unclassified error | Stop and return the error. | @@ -194,6 +195,15 @@ Failover is intentionally bounded. It helps with target-specific availability, a quota, and overload failures; it does not hide caller errors or policy refusals. ::: +For streaming requests, the upstream HTTP status is not the final decision. OpenCodex buffers a +bounded pre-output prefix of the selected child's Responses SSE. If the stream reports a retryable +`response.failed` terminal before any text, reasoning, tool call, or other output event, the child +is recorded as failed and the combo may try its next eligible target. Once any output event begins, +the target is committed: a later stream failure is returned to the client and is never replayed on +another provider, which prevents duplicate text and tool execution. If the pre-output buffer reaches +its safety cap without a terminal or output boundary, OpenCodex also commits the current target +instead of growing memory without a bound. + ## Default reasoning effort `defaultEffort` supplies `reasoning.effort` only when all of these are true: diff --git a/docs-site/src/content/docs/ja/guides/combos.md b/docs-site/src/content/docs/ja/guides/combos.md index 7cad2b0b6e..6387c64392 100644 --- a/docs-site/src/content/docs/ja/guides/combos.md +++ b/docs-site/src/content/docs/ja/guides/combos.md @@ -108,6 +108,7 @@ ocx combo set balanced \ |結果 |行動 | | --- | --- | | HTTP 401、403、404、408、429、または任意の 5xx |ターゲットを冷却し、次の適格なターゲットに移動します。 | +|モデルのサポート終了、retired、deprecated、sunset、decommissioned、または利用不可を明示する HTTP 410 |そのターゲットをクールダウンし、次へ進みます。無関係な 410 はターミナル エラーのままです。 | |機密認証、サブスクリプション、クォータ、レート制限、過負荷、またはアップストリーム サーバー エラー |ステータスだけでは物足りない場合でもターゲットを冷やしてホップさせましょう。 | |クライアントのキャンセル (499)、`origin_rejected`、サイバー ポリシーの拒否、コンテキスト オーバーフロー、または無効なリクエスト |停止してエラーを返します。別のターゲットではリクエストは有効になりません。 | |その他の未分類のエラー |停止してエラーを返します。 | @@ -120,6 +121,8 @@ ocx combo set balanced \ フェイルオーバーは意図的に制限されています。これは、ターゲット固有の可用性、認証、クォータ、および過負荷の障害に役立ちます。呼び出し元のエラーやポリシーの拒否は隠蔽されません。 ::: +ストリーミング リクエストでは、アップストリームの HTTP ステータスだけで最終判断しません。OpenCodex は、選択した子ターゲットの Responses SSE を出力開始前の上限付き範囲だけバッファします。テキスト、推論、ツール呼び出し、またはその他の出力イベントが始まる前に再試行可能な `response.failed` ターミナルを受け取った場合、その子を失敗として記録し、次の適格なターゲットを試せます。出力が始まるかバッファ上限に達した時点で現在のターゲットにコミットし、その後のストリーム失敗を別プロバイダーへ再送しません。これによりテキストやツール実行の重複を防ぎます。 + ## デフォルトの推論負荷 `defaultEffort` は、次のすべてが当てはまる場合にのみ `reasoning.effort` を提供します。 diff --git a/docs-site/src/content/docs/ko/guides/combos.md b/docs-site/src/content/docs/ko/guides/combos.md index cec9ed6258..20402a5802 100644 --- a/docs-site/src/content/docs/ko/guides/combos.md +++ b/docs-site/src/content/docs/ko/guides/combos.md @@ -108,6 +108,7 @@ ocx combo set balanced \ | 결과 | 동작 | | --- | --- | | HTTP 401, 403, 404, 408, 429, 또는 모든 5xx | 대상을 쿨다운으로 보내고 다음 적합한 대상으로 넘어갑니다. | +| 모델 수명 종료, retired, deprecated, sunset, decommissioned, 또는 더 이상 사용할 수 없다는 신호가 명시된 HTTP 410 | 해당 대상만 쿨다운으로 보내고 다음 대상으로 넘어갑니다. 관련 없는 410은 종결 오류로 유지합니다. | | 인증, 구독, 쿼터, 속도 제한, 과부하, 또는 상위 서버 오류로 분류됨 | 상태 코드만으로는 충분하지 않더라도 대상을 쿨다운으로 보내고 넘어갑니다. | | 클라이언트 취소(499), `origin_rejected`, cyber-policy refusal, context overflow, 또는 invalid request | 멈추고 오류를 반환합니다. 다른 대상을 써도 요청이 유효해지지 않기 때문입니다. | | 그 밖의 분류되지 않은 오류 | 멈추고 오류를 반환합니다. | @@ -120,6 +121,8 @@ ocx combo set balanced \ 페일오버는 의도적으로 범위를 제한합니다. 대상별 가용성, 인증, 쿼터, 과부하 실패에는 도움이 되지만, 호출자 오류나 정책 거부를 숨기지는 않습니다. ::: +스트리밍 요청에서는 상위 HTTP 상태만으로 최종 결정을 내리지 않습니다. OpenCodex는 선택한 하위 대상의 Responses SSE를 출력 시작 전의 제한된 구간까지만 버퍼링합니다. 텍스트, 추론, 도구 호출 또는 그 밖의 출력 이벤트가 시작되기 전에 재시도 가능한 `response.failed` 종결 이벤트가 오면 해당 시도를 실패로 기록하고 다음 적합한 대상을 시도할 수 있습니다. 출력이 시작되거나 버퍼 상한에 도달하면 현재 대상에 커밋하며, 이후의 스트림 실패를 다른 공급자에서 다시 실행하지 않습니다. 따라서 텍스트와 도구 실행이 중복되지 않습니다. + ## 기본 reasoning effort `defaultEffort`는 다음 조건이 모두 참일 때만 `reasoning.effort`를 채웁니다. diff --git a/docs-site/src/content/docs/ru/guides/combos.md b/docs-site/src/content/docs/ru/guides/combos.md index b2baf981cc..d68f9c2b8e 100644 --- a/docs-site/src/content/docs/ru/guides/combos.md +++ b/docs-site/src/content/docs/ru/guides/combos.md @@ -135,6 +135,7 @@ ocx combo set balanced \ | Результат | Поведение | | --- | --- | | HTTP 401, 403, 404, 408, 429, или любой 5xx | Перевести цель в cooldown и перейти к следующей подходящей цели. | +| HTTP 410 с явным признаком окончания срока службы модели, retirement, deprecated, sunset, decommissioned или недоступности | Перевести только эту цель в cooldown и перейти дальше. Несвязанные ответы 410 остаются terminal-ошибками. | | Классифицированная ошибка аутентификации, подписки, квоты, rate-limit, перегрузки или upstream-server | Перевести цель в cooldown и переключиться, даже если одного статуса недостаточно. | | Отмена клиентом (499), `origin_rejected`, отказ из-за cyber-policy, переполнение контекста или некорректный запрос | Остановиться и вернуть ошибку; другая цель не сделает такой запрос корректным. | | Любая другая неклассифицированная ошибка | Остановиться и вернуть ошибку. | @@ -152,6 +153,8 @@ Failover намеренно ограничен. Он помогает при п аутентификации, квоты и перегрузки; он не скрывает ошибки вызывающей стороны и отказы политики. ::: +Для потоковых запросов одного HTTP-статуса upstream недостаточно для окончательного решения. OpenCodex буферизует только ограниченный префикс Responses SSE выбранной дочерней цели до начала вывода. Если повторяемый terminal `response.failed` приходит до текста, reasoning, вызова инструмента или другого события вывода, попытка отмечается как неудачная и combo может перейти к следующей подходящей цели. После начала вывода или достижения лимита буфера текущая цель считается выбранной; более поздний сбой потока не воспроизводится у другого провайдера. Это предотвращает дублирование текста и выполнения инструментов. + ## Effort по умолчанию `defaultEffort` подставляет `reasoning.effort` только если одновременно выполняются все условия: diff --git a/docs-site/src/content/docs/zh-cn/guides/combos.md b/docs-site/src/content/docs/zh-cn/guides/combos.md index fe89fc04c7..54180df9b4 100644 --- a/docs-site/src/content/docs/zh-cn/guides/combos.md +++ b/docs-site/src/content/docs/zh-cn/guides/combos.md @@ -134,6 +134,7 @@ combo 失败分为 **跳转** 失败和 **终止** 失败。 | 结果 | 行为 | | --- | --- | | HTTP 401、403、404、408、429,或任何 5xx | 使该目标进入冷却,并跳转到下一个合格目标。 | +| HTTP 410,并明确表明模型已到生命周期终点、retired、deprecated、sunset、decommissioned 或不再可用 | 仅冷却该目标并继续跳转。无关的 410 仍然是终止错误。 | | 被分类为认证、订阅、配额、速率限制、过载或上游服务器错误 | 即使仅凭状态码不足以判断,也会使该目标进入冷却并跳转。 | | 客户端取消(499)、`origin_rejected`、cyber-policy 拒绝、上下文溢出,或无效请求 | 停止并返回错误;换其他目标也无法让请求变得有效。 | | 任何其他未分类错误 | 停止并返回错误。 | @@ -146,6 +147,8 @@ combo 失败分为 **跳转** 失败和 **终止** 失败。 故障切换是有边界的。它有助于处理特定目标的可用性、认证、配额和过载失败;它不会掩盖调用方错误或策略拒绝。 ::: +对于流式请求,上游 HTTP 状态并不是最终决定。OpenCodex 只会缓冲所选子目标在开始输出前的一段有上限的 Responses SSE。若在任何文本、推理、工具调用或其他输出事件开始之前收到可重试的 `response.failed` 终止事件,该次尝试会被记为失败,combo 可以继续尝试下一个合格目标。一旦输出开始或预输出缓冲区达到上限,当前目标就会被提交;之后的流错误不会在其他提供商上重放,从而避免重复文本和重复执行工具。 + ## 默认推理力度 只有在以下所有条件都满足时,`defaultEffort` 才会提供 `reasoning.effort`: diff --git a/src/combos/failover.ts b/src/combos/failover.ts index dc3578d0a3..97bd1f5928 100644 --- a/src/combos/failover.ts +++ b/src/combos/failover.ts @@ -109,6 +109,28 @@ export function clearComboTargetCooldowns(comboId?: string): void { export type ComboFailureDecision = "hop" | "stop"; +function isModelLifecycleGone( + status: number, + message: string, + code?: string | null, +): boolean { + if (status !== 410) return false; + const normalizedCode = code?.trim().toLowerCase().replaceAll("-", "_"); + if ([ + "model_deprecated", + "model_end_of_life", + "model_eol", + "model_not_found", + "model_retired", + ].includes(normalizedCode ?? "")) return true; + const text = message.toLowerCase(); + return /\bmodel\b/.test(text) && ( + /\bend[ -]of[ -]life\b/.test(text) + || /\bno longer available\b/.test(text) + || /\b(?:deprecated|retired|retirement|sunset|decommissioned)\b/.test(text) + ); +} + export function comboFailureDecision( status: number, message: string, @@ -119,6 +141,11 @@ export function comboFailureDecision( // Cyber policy is a hard non-retryable refusal — honor structured code even when // classificationText was truncated before the JSON code field. if (isCyberPolicyCode(options?.code)) return "stop"; + // HTTP 410 is normally terminal. A model-specific lifecycle verdict is target-local, + // however: another provider/model in the declared combo can still serve the request. + // Require structured lifecycle code or explicit model+lifecycle prose so unrelated + // application-level 410 responses remain fail-closed. + if (isModelLifecycleGone(status, message, options?.code)) return "hop"; const error = classifyError(status, "upstream_error", message); if (isCyberPolicyCode(error.code)) return "stop"; // A local input-admission refusal (#1524) says "this candidate cannot fit the request", diff --git a/src/server/responses/combo-stream-preflight.ts b/src/server/responses/combo-stream-preflight.ts new file mode 100644 index 0000000000..5f06bfeff5 --- /dev/null +++ b/src/server/responses/combo-stream-preflight.ts @@ -0,0 +1,171 @@ +import type { ResponsesTerminalStatus } from "../../bridge"; +import type { RequestLogContext } from "../request-log"; +import { createSseInspector } from "../relay"; +import { MAX_CLIENT_SSE_FRAME_BYTES } from "../sse-frame-buffer"; + +const COMBO_STREAM_PREFLIGHT_MAX_BYTES = MAX_CLIENT_SSE_FRAME_BYTES; + +const PRE_OUTPUT_CONTROL_EVENTS = new Set([ + "response.created", + "response.in_progress", + "response.queued", + "response.heartbeat", +]); + +const TERMINAL_EVENTS = new Set([ + "response.completed", + "response.failed", + "response.incomplete", +]); + +/** + * Decide when replaying the request on another combo target would risk duplicating + * client-visible output or a tool-side effect. Unknown event types commit the child + * conservatively; only the small Responses lifecycle preamble remains replayable. + */ +export function comboStreamPayloadCommitsOutput(payload: unknown): boolean { + if (!payload || typeof payload !== "object" || Array.isArray(payload)) return true; + const type = (payload as { type?: unknown }).type; + if (typeof type !== "string") return true; + return !PRE_OUTPUT_CONTROL_EVENTS.has(type) && !TERMINAL_EVENTS.has(type); +} + +function replayBufferedResponse( + response: Response, + reader: ReadableStreamDefaultReader, + buffered: Uint8Array[], +): Response { + let index = 0; + const body = new ReadableStream({ + async pull(controller) { + if (index < buffered.length) { + controller.enqueue(buffered[index++]!); + return; + } + try { + const next = await reader.read(); + if (next.done) controller.close(); + else controller.enqueue(next.value); + } catch (error) { + try { controller.error(error); } catch { /* consumer already closed */ } + } + }, + cancel(reason) { + reader.cancel(reason).catch(() => undefined); + }, + }); + return new Response(body, { + status: response.status, + statusText: response.statusText, + headers: response.headers, + }); +} + +function failedTerminalResponse( + response: Response, + terminalPayload: Record, + logCtx: RequestLogContext, +): Response { + const nested = terminalPayload.response; + const terminalResponse = nested && typeof nested === "object" && !Array.isArray(nested) + ? nested as Record + : {}; + const nestedError = terminalResponse.error; + const error = nestedError && typeof nestedError === "object" && !Array.isArray(nestedError) + ? nestedError as Record + : { + type: "upstream_error", + code: "upstream_server_error", + message: logCtx.upstreamError ?? "Provider stream failed before producing output", + }; + const headers = new Headers(response.headers); + headers.set("content-type", "application/json"); + headers.delete("content-length"); + headers.delete("content-encoding"); + const usage = terminalResponse.usage; + return new Response(JSON.stringify({ + error, + // The combo classifier needs only the error and optional usage. Do not carry + // response ids, provider metadata, or future terminal fields into the client + // error envelope merely because they shared the terminal snapshot. + response: { + error, + ...(usage && typeof usage === "object" && !Array.isArray(usage) ? { usage } : {}), + }, + }), { + status: logCtx.terminalHttpStatus ?? 502, + headers, + }); +} + +export type ComboStreamPreflightResult = + | { kind: "accepted"; response: Response } + | { kind: "failed"; response: Response }; + +/** + * Buffer a combo child's downstream SSE only until the request becomes unsafe to + * replay or reaches a terminal. This owns exactly one body reader. The aggregate + * buffer is capped; hitting the cap commits the current target instead of growing + * memory or guessing that replay is safe. + */ +export async function preflightComboStreamResponse( + response: Response, + logCtx: RequestLogContext, +): Promise { + const contentType = response.headers.get("content-type")?.toLowerCase() ?? ""; + if (!response.ok || !response.body || !contentType.includes("text/event-stream")) { + return { kind: "accepted", response }; + } + + const reader = response.body.getReader(); + const buffered: Uint8Array[] = []; + let bufferedBytes = 0; + let outputCommitted = false; + let terminalStatus: ResponsesTerminalStatus | undefined; + let failedPayload: Record | undefined; + const inspector = createSseInspector({ + logCtx, + onParsedPayload: payload => { + if (comboStreamPayloadCommitsOutput(payload)) outputCommitted = true; + if (!payload || typeof payload !== "object" || Array.isArray(payload)) return; + if ((payload as { type?: unknown }).type === "response.failed") { + failedPayload = payload as Record; + } + }, + onTerminal: status => { terminalStatus = status; }, + }); + + try { + for (;;) { + const next = await reader.read(); + if (next.done) { + inspector.finish(); + } else { + if (bufferedBytes + next.value.byteLength > COMBO_STREAM_PREFLIGHT_MAX_BYTES) { + // Keep the cap about memory the preflight allocates. The upstream chunk already exists; + // copying it before committing would transiently exceed the boundary for no + // replay benefit. Preserve it unsliced behind the already-bounded prefix. + return { + kind: "accepted", + response: replayBufferedResponse(response, reader, [...buffered, next.value]), + }; + } + const retained = next.value.slice(); + buffered.push(retained); + bufferedBytes += retained.byteLength; + inspector.feed(retained); + } + + if (terminalStatus === "failed" && !outputCommitted && failedPayload) { + await reader.cancel("retrying zero-output combo stream failure").catch(() => undefined); + return { kind: "failed", response: failedTerminalResponse(response, failedPayload, logCtx) }; + } + if (next.done || terminalStatus !== undefined || outputCommitted + || bufferedBytes >= COMBO_STREAM_PREFLIGHT_MAX_BYTES) { + return { kind: "accepted", response: replayBufferedResponse(response, reader, buffered) }; + } + } + } finally { + inspector.dispose(); + } +} diff --git a/src/server/responses/core.ts b/src/server/responses/core.ts index 8e6afce616..f1081e4fe3 100644 --- a/src/server/responses/core.ts +++ b/src/server/responses/core.ts @@ -325,6 +325,12 @@ import { emptyCompletionRetryEnabled, guardEmptyCompletionEventStream, } from "./empty-completion-guard"; +import { preflightComboStreamResponse } from "./combo-stream-preflight"; + +// runTurn adapters own an event queue and perform their combo preflight before +// bridging. A second byte-stream reader would reinterpret that transport's +// already-committed event boundary and can replay custom adapter work. +const runTurnAdapterSseResponses = new WeakSet(); /** * Adapters whose continuation state must survive Codex's store:false requests. @@ -1874,6 +1880,7 @@ export async function handleComboResponses( }); let resolvedAuth: CodexAuthContext | undefined; let terminalRecorder: ((status: ResponsesTerminalStatus, httpStatusOverride?: number) => void) | undefined; + let terminalOutcomeRecorded = false; const started = Date.now(); const attempt = beginRequestAttempt( (logCtx.attempts?.length ?? 0) + 1, @@ -1920,7 +1927,15 @@ export async function handleComboResponses( options.onFirstOutput?.(); }, onCodexAuthContextResolved: value => { resolvedAuth = value; }, - setTerminalOutcomeRecorder: value => { terminalRecorder = value; }, + setTerminalOutcomeRecorder: value => { + terminalRecorder = value + ? (status, httpStatusOverride) => { + if (terminalOutcomeRecorded) return; + terminalOutcomeRecorded = true; + value(status, httpStatusOverride); + } + : undefined; + }, onConsumedComboFailure: value => { consumedChildFailure = value; }, onNativePassthroughTerminal: callbackGate.onTerminal, onNativePassthroughCancel: callbackGate.onCancel, @@ -1940,6 +1955,31 @@ export async function handleComboResponses( return clientCancelledResponse(); } + if (response.ok && !runTurnAdapterSseResponses.has(response)) { + const nativePassthrough = isNativePassthroughSseResponse(response); + const eagerRelay = isEagerRelaySseResponse(response); + let preflight; + try { + preflight = await preflightComboStreamResponse(response, childLog); + } catch (error) { + callbackGate.discard(); + if (options.abortSignal?.aborted) { + retainCancelledAttempt(); + return clientCancelledResponse(); + } + throw error; + } + if (preflight.kind === "failed") { + callbackGate.discard(); + terminalRecorder?.("failed", preflight.response.status); + response = preflight.response; + } else { + response = preflight.response; + if (nativePassthrough) markNativePassthroughSseResponse(response); + if (eagerRelay) markEagerRelaySseResponse(response); + } + } + if (response.ok) { sealRequestAttemptIdentity( attempt, @@ -1995,7 +2035,7 @@ export async function handleComboResponses( ); finishRequestAttempt( attempt, - response.status, + failure.response.status, Date.now() - started, failure.usage, ); @@ -2009,7 +2049,7 @@ export async function handleComboResponses( return lastFailure; } console.warn( - `[combo] ${comboId}: ${targetKey(pick.target)} failed with ${response.status} after ${Date.now() - started}ms`, + `[combo] ${comboId}: ${targetKey(pick.target)} failed with ${failure.response.status} after ${Date.now() - started}ms`, ); const nextPick = advanceComboAfterFailure(config, pick, { retryAfter: failure.retryAfter, @@ -4438,9 +4478,11 @@ async function handleResponsesInner( ); const bridgeTurnAc = new AbortController(); const trackedSse = trackStreamLifetime(sseStream, bridgeTurnAc, undefined, options.turnAdmissionLease); - return new Response(trackedSse, { + const response = new Response(trackedSse, { headers: { "Content-Type": "text/event-stream", "Cache-Control": "no-cache", "Connection": "keep-alive", "X-Accel-Buffering": "no" }, }); + runTurnAdapterSseResponses.add(response); + return response; } await runTurn(); diff --git a/structure/04_transports-and-sidecars.md b/structure/04_transports-and-sidecars.md index eca817d9a4..a81cae7acd 100644 --- a/structure/04_transports-and-sidecars.md +++ b/structure/04_transports-and-sidecars.md @@ -1142,6 +1142,39 @@ combo whose remaining eligible targets use other providers. - 장점, 단점 및 영향: Same-account model fallback works without weakening explicit upstream backoff; the account health map intentionally does not remember that one deferred reset-derived failure, while the combo target map does. ``` +## Combo streaming commit boundary + +An HTTP 200 does not by itself commit a streaming combo child. The combo parent runs the child's +downstream Responses SSE through `src/server/responses/combo-stream-preflight.ts`, which owns one +reader and buffers only until one of these boundaries: + +- a non-control Responses event begins client-visible output or a tool/action item, after which the + target is committed and cross-target replay is forbidden; +- a `response.failed` terminal arrives first, in which case the terminal is converted back through + the ordinary bounded combo-failure classifier and may advance to the next declared target; +- a completed/incomplete terminal or the aggregate preflight byte cap is reached, in which case the + current target is committed conservatively. + +The buffered bytes are replayed unchanged before the reader continues. Native passthrough and eager +relay identity markers are restored on the wrapped response so Windows/Bun stream paths and deferred +logging retain their existing owners. A failed child keeps its physical attempt receipt and usage, +while the successful child remains the logical request result. + +HTTP 410 remains terminal by default. It advances and cools only the exact combo target when the +structured code or message explicitly identifies a model lifecycle event (end-of-life, retired, +deprecated, sunset, decommissioned, or no longer available). An unrelated application-level 410 is +not retried. + +```text +[Decision Log] +- 목적과 의도: Recover a failover combo from a provider-local SSE or model-lifecycle failure only while replay is provably free of duplicate client output and tool calls. +- 기존 구현 및 제약 조건: The parent committed every HTTP-200 child before reading its SSE body, while terminal stream errors were classified only later by logging; generic 410 responses stopped the chain. +- 검토한 주요 대안: Retry every failed stream, buffer the complete turn, inspect only HTTP status, or preflight a bounded prefix until an explicit output/terminal boundary. +- 선택한 방식: Put the one-reader bounded preflight in a dedicated module, commit on any non-control event, and treat only explicit model-lifecycle 410 evidence as target-local. +- 다른 대안 대신 이 방식을 선택한 이유: Replaying after output can duplicate text or tools, full-turn buffering destroys streaming and grows memory, and making every 410 retryable hides caller/application errors. +- 장점, 단점 및 영향: Zero-output provider failures can reach a healthy target with ordered receipts and cooldown; ambiguous or oversized pre-output streams keep the current fail-closed behavior instead of consuming unbounded memory. +``` + ## Transport inventory The sections above cover the transports with load-bearing invariants. The rest of the transport diff --git a/tests/combo-stream-preflight.test.ts b/tests/combo-stream-preflight.test.ts new file mode 100644 index 0000000000..7727597174 --- /dev/null +++ b/tests/combo-stream-preflight.test.ts @@ -0,0 +1,93 @@ +import { describe, expect, test } from "bun:test"; +import { + comboStreamPayloadCommitsOutput, + preflightComboStreamResponse, +} from "../src/server/responses/combo-stream-preflight"; +import type { RequestLogContext } from "../src/server/request-log"; +import { MAX_CLIENT_SSE_FRAME_BYTES } from "../src/server/sse-frame-buffer"; + +const sse = (...payloads: unknown[]): Response => new Response( + payloads.map(payload => `data: ${JSON.stringify(payload)}\n\n`).join(""), + { headers: { "content-type": "text/event-stream" } }, +); + +describe("combo stream preflight", () => { + test("keeps only lifecycle preamble replayable and treats unknown output conservatively", () => { + expect(comboStreamPayloadCommitsOutput({ type: "response.created" })).toBe(false); + expect(comboStreamPayloadCommitsOutput({ type: "response.heartbeat" })).toBe(false); + expect(comboStreamPayloadCommitsOutput({ type: "response.failed" })).toBe(false); + expect(comboStreamPayloadCommitsOutput({ type: "response.output_text.delta", delta: "x" })).toBe(true); + expect(comboStreamPayloadCommitsOutput({ type: "response.output_item.added", item: { type: "function_call" } })).toBe(true); + expect(comboStreamPayloadCommitsOutput({ type: "provider.future_event" })).toBe(true); + }); + + test("converts a zero-output failed terminal into a retryable HTTP failure", async () => { + const logCtx: RequestLogContext = { model: "m1", provider: "a" }; + const result = await preflightComboStreamResponse(sse( + { type: "response.created", response: { id: "r1", status: "in_progress" } }, + { + type: "response.failed", + response: { + id: "r1", + status: "failed", + error: { type: "server_error", code: "upstream_server_error", message: "busy" }, + usage: { input_tokens: 7, output_tokens: 0, total_tokens: 7 }, + provider_trace_id: "must-not-cross-the-combo-boundary", + }, + }, + ), logCtx); + + expect(result.kind).toBe("failed"); + expect(result.response.status).toBe(502); + const body = await result.response.json(); + expect(body).toMatchObject({ + error: { code: "upstream_server_error", message: "busy" }, + response: { usage: { input_tokens: 7, output_tokens: 0 } }, + }); + expect(JSON.stringify(body)).not.toContain("provider_trace_id"); + }); + + test("replays buffered bytes unchanged after output commits the target", async () => { + const original = [ + { type: "response.created", response: { id: "r1", status: "in_progress" } }, + { type: "response.output_text.delta", delta: "visible" }, + { + type: "response.failed", + response: { status: "failed", error: { type: "server_error", message: "late" } }, + }, + ]; + const source = sse(...original); + const expected = await source.clone().text(); + const result = await preflightComboStreamResponse(source, { model: "m1", provider: "a" }); + + expect(result.kind).toBe("accepted"); + expect(await result.response.text()).toBe(expected); + }); + + test("commits an oversized next chunk without copying it beyond the preflight cap", async () => { + const encoder = new TextEncoder(); + const preamble = encoder.encode(`data: ${JSON.stringify({ + type: "response.created", + response: { id: "r1", status: "in_progress" }, + })}\n\n`); + const oversized = new Uint8Array(MAX_CLIENT_SSE_FRAME_BYTES + 1); + oversized.fill(120); + const chunks = [preamble, oversized]; + const response = new Response(new ReadableStream({ + pull(controller) { + const chunk = chunks.shift(); + if (chunk) controller.enqueue(chunk); + else controller.close(); + }, + }), { headers: { "content-type": "text/event-stream" } }); + + const result = await preflightComboStreamResponse(response, { model: "m1", provider: "a" }); + expect(result.kind).toBe("accepted"); + const reader = result.response.body!.getReader(); + const first = await reader.read(); + const second = await reader.read(); + expect(new TextDecoder().decode(first.value)).toBe(new TextDecoder().decode(preamble)); + expect(second.value).toBe(oversized); + await reader.cancel(); + }); +}); diff --git a/tests/combos.test.ts b/tests/combos.test.ts index 70bf3a657b..166a738554 100644 --- a/tests/combos.test.ts +++ b/tests/combos.test.ts @@ -12,7 +12,6 @@ import { comboDefaultEffort, comboDisabledModelId, comboDisabledModelSelectors, - comboFailureDecision, comboIdFromRawBody, comboModelId, comboPublicModelId, @@ -37,6 +36,7 @@ import { tryPickComboModel, UnknownComboError, } from "../src/combos"; +import { comboFailureDecision } from "../src/combos/failover"; import { getConfigPath, readConfigDiagnostics, saveConfig } from "../src/config"; import { routeModel } from "../src/router"; import { handleManagementAPI } from "../src/server/management-api"; @@ -355,6 +355,10 @@ describe("combo failure policy and advancement", () => { expect(comboFailureDecision(403, '{"code":"origin_rejected"}')).toBe("stop"); expect(comboFailureDecision(413, "request too large")).toBe("stop"); expect(comboFailureDecision(409, "conflict")).toBe("stop"); + expect(comboFailureDecision(410, "resource is gone")).toBe("stop"); + expect(comboFailureDecision(410, "The model has reached its end of life and is no longer available.")).toBe("hop"); + expect(comboFailureDecision(410, "The model is scheduled for retirement.")).toBe("hop"); + expect(comboFailureDecision(410, "gone", { code: "model_retired" })).toBe("hop"); expect(comboFailureDecision(499, "client cancelled")).toBe("stop"); expect(comboFailureDecision(422, "invalid_api_key")).toBe("hop"); // #1524: a LOCAL input-admission refusal means "this candidate cannot fit the request", diff --git a/tests/server-combo-failover-e2e.test.ts b/tests/server-combo-failover-e2e.test.ts index 81b96f4834..415419c560 100644 --- a/tests/server-combo-failover-e2e.test.ts +++ b/tests/server-combo-failover-e2e.test.ts @@ -202,6 +202,17 @@ function chatStream(text: string): Response { return new Response(frames, { headers: { "content-type": "text/event-stream" } }); } +function chatErrorStream(message: string, prefix?: string): Response { + const frames = [ + ...(prefix + ? [`data: ${JSON.stringify({ choices: [{ index: 0, delta: { content: prefix }, finish_reason: null }] })}\n\n`] + : []), + `data: ${JSON.stringify({ error: { type: "server_error", code: "upstream_server_error", message } })}\n\n`, + "data: [DONE]\n\n", + ].join(""); + return new Response(frames, { headers: { "content-type": "text/event-stream" } }); +} + function responsesSuccess(text: string, model = "responses-model"): Record { return { id: `resp-${model}`, @@ -447,6 +458,109 @@ describe("server combo failover 030 activation matrix", () => { expect(hits).toEqual(["a:false", "b:false", "a:true", "b:true"]); }); + test("zero-output terminal SSE failure hops before committing the child stream", async () => { + const hits: string[] = []; + const a = serve(() => { + hits.push("a"); + return chatErrorStream("service busy, please try again later"); + }); + const b = serve(() => { + hits.push("b"); + return chatStream("stream backup"); + }); + const config = comboConfig({ + a: provider("openai-chat", baseUrl(a), "key-a"), + b: provider("openai-chat", baseUrl(b), "key-b"), + }); + + const response = await postLogged(config, { stream: true }); + expect(response.status).toBe(200); + expect(JSON.stringify(await collectSse(response))).toContain("stream backup"); + expect(hits).toEqual(["a", "b"]); + + const { log, usage } = await latestAttemptReceipts(config); + for (const receipt of [log, usage]) { + expect(receipt).toMatchObject({ + provider: "combo", + model: "combo/free", + resolvedModel: "m2", + attempts: [ + { ordinal: 1, provider: "a", model: "m1", status: 502 }, + { ordinal: 2, provider: "b", model: "m2", status: 200 }, + ], + }); + expect(receipt.attempts[0]).not.toHaveProperty("firstOutputMs"); + } + }); + + test("terminal SSE failure after output stays on the first target and never replays", async () => { + const hits: string[] = []; + const a = serve(() => { + hits.push("a"); + return chatErrorStream("late service failure", "already visible"); + }); + const b = serve(() => { + hits.push("b"); + return chatStream("must not replay"); + }); + const config = comboConfig({ + a: provider("openai-chat", baseUrl(a), "key-a"), + b: provider("openai-chat", baseUrl(b), "key-b"), + }); + + const response = await post(config, { stream: true }); + expect(response.status).toBe(200); + const frames = await collectSse(response); + expect(JSON.stringify(frames)).toContain("already visible"); + expect(frames.some(frame => frame.data.type === "response.failed")).toBe(true); + expect(JSON.stringify(frames)).not.toContain("must not replay"); + expect(hits).toEqual(["a"]); + }); + + test("model-lifecycle 410 hops once and cools only the dead combo target", async () => { + const hits: string[] = []; + const a = serve(() => { + hits.push("a"); + return Response.json({ + error: { + type: "invalid_request_error", + code: "model_end_of_life", + message: "The model 'm1' has reached its end of life and is no longer available.", + }, + }, { status: 410 }); + }); + const b = serve(() => { + hits.push("b"); + return chatSuccess("lifecycle backup", "m2"); + }); + const targets = [ + { provider: "a", model: "m1" }, + { provider: "b", model: "m2" }, + ]; + const config = comboConfig({ + a: provider("openai-chat", baseUrl(a), "key-a"), + b: provider("openai-chat", baseUrl(b), "key-b"), + }, targets); + + const response = await postLogged(config); + expect(response.status).toBe(200); + expect(await response.text()).toContain("lifecycle backup"); + expect(isComboTargetInCooldown("free", targets[0]!)).toBe(true); + const { log, usage } = await latestAttemptReceipts(config); + for (const receipt of [log, usage]) { + expect(receipt.attempts).toMatchObject([ + { ordinal: 1, provider: "a", model: "m1", status: 410 }, + { ordinal: 2, provider: "b", model: "m2", status: 200 }, + ]); + } + + clearComboSelectionState(); + const retry = await post(config); + expect(retry.status).toBe(200); + expect(await retry.text()).toContain("lifecycle backup"); + expect(hits).toEqual(["a", "b", "b"]); + }); + test("persists one logical A503 to B200 request with ordered physical usage", async () => { const a = serve(() => Response.json({ error: { message: "overloaded" }, @@ -2260,6 +2374,32 @@ describe("server combo failover 030 activation matrix", () => { expect(frames.some(frame => frame.event === "response.completed")).toBe(false); }); + test("runTurn control-only late errors stay on the adapter-owned stream", async () => { + let aHits = 0; + let bHits = 0; + customRunTurn = async (_parsed, _incoming, emit) => { + aHits += 1; + // preflightAdapterEvents commits this custom transport at its first + // non-heartbeat event even though the bridge emits no visible output. + emit({ type: "assistant_boundary" }); + emit({ type: "error", message: "late runTurn failure" }); + }; + const b = serve(() => { + bHits += 1; + return chatStream("must not replay"); + }); + const config = comboConfig({ + a: provider("test-run-turn", "test://run-turn", "key-a"), + b: provider("openai-chat", baseUrl(b), "key-b"), + }); + const response = await post(config, { stream: true }); + const frames = await collectSse(response); + expect(aHits).toBe(1); + expect(bHits).toBe(0); + expect(frames.filter(frame => frame.event === "response.failed")).toHaveLength(1); + expect(frames.some(frame => frame.event === "response.output_text.delta")).toBe(false); + }); + test("PATCH-disable-all returns combo_unavailable without any fallback hit", async () => { let aHits = 0; let bHits = 0;