[Bug]: failover combos stop on zero-output SSE terminal failures and model-EOL HTTP 410 responses

[FacuM]

Client or integration

Codex App

Area

Proxy and routing

Summary

A strategy: "failover" combo can stop without trying other eligible targets in two target-specific failure cases:

1. An OpenAI-compatible provider returns HTTP 200 for a streaming request, then emits a terminal service-busy error before any assistant output. OpenCodex records the attempt as 502 / upstream_server_error, but the combo does not advance to the next eligible target.
2. A provider returns HTTP 410 because the selected model reached end of life. OpenCodex returns the 410 to Codex App and repeatedly selects the same dead target, although several other combo targets are eligible.

In both captured cases firstOutputMs was null, so no meaningful assistant output had been committed and bounded replay to the next target should have been safe.

The first upstream incident was described as a service-busy/503 condition inside an HTTP-200 SSE stream. OpenCodex normalized the terminal stream event to HTTP 502 in its usage log.

Expected behavior:

• A terminal provider error received before the first meaningful assistant output should be eligible for bounded combo failover, including when the upstream HTTP status was 200 and the failure arrived inside SSE.
• Once any text/tool output has been emitted, OpenCodex should continue to fail closed rather than replaying and risking duplicate output or tool calls.
• HTTP 410 with a provider/model lifecycle signal should advance to the next combo target. The dead target should receive an appropriate cooldown/circuit-breaker state so immediate client retries do not select it again.
• Attempts should remain capped by the declared target list and retain per-attempt usage/routing traces.

Current 2.31.0 behavior matches two source gaps:

• src/server/responses/core.ts accepts response.ok at the combo boundary before the OpenAI-compatible SSE stream is parsed. A later terminal stream failure updates the request log, but the parent combo loop has already exited.
• comboFailureDecision(410, "...end of life...") returns "stop" in src/combos/failover.ts.

The same paths are still present in release 2.31.0 (6ae83b1f189c353935d4977bb01227484fbdb52b) and current dev inspected at 4f41a8e936141af7ee828e335da314b9dc1ef761:

• https://github.com/lidge-jun/opencodex/blob/6ae83b1f189c353935d4977bb01227484fbdb52b/src/combos/failover.ts#L112-L165

• opencodex/src/server/responses/core.ts

  Lines 1903 to 1928 in 6ae83b1

  	if (response.ok) {
  	sealRequestAttemptIdentity(
  	attempt,
  	childLog.provider,
  	childLog.providerAdapter ?? attempt.adapter,
  	childLog.accountLogLabel,
  	);
  	(logCtx.attempts ??= []).push(attempt);
  	attemptRetained = true;
  	noteComboSuccess(comboId, combo, pick.target, pick.writerGeneration);
  	Object.assign(logCtx, childLog, {
  	requestedModel,
  	model: requestedModel,
  	provider: "combo",
  	comboId,
  	routeDecision: logCtx.routeDecision,
  	attempts: logCtx.attempts,
  	activeAttempt: attempt,
  	activeAttemptStartedAt: started,
  	resolvedModel: childLog.resolvedModel ?? childLog.model,
  	});
  	options.onCodexAuthContextResolved?.(resolvedAuth);
  	options.setTerminalOutcomeRecorder?.(terminalRecorder);
  	callbackGate.commit();
  	return response;
  	}

• opencodex/src/server/responses/core.ts

  Lines 5136 to 5179 in 6ae83b1

  	if (parsed.stream) {
  	const initialEventStream = activeAdapter.parseStream(
  	upstreamResponse,
  	translatorBudget,
  	logCtx.activeTierMetadata,
  	);
  	const eventStream = terminalGuardEnabled
  	? guardTerminalEventStream({
  	parsed,
  	firstEvents: initialEventStream,
  	adapterName: activeAdapter.name,
  	maxAutoContinuations: 1,
  	continuation: fetchTerminalGuardContinuation,
  	})
  	: initialEventStream;
  	// The empty-completion guard sits OUTSIDE the terminal guard: a completed
  	// turn with no text and no tool call is retried with the IDENTICAL request
  	// (fetchTerminalGuardContinuation(parsed) replays the cached byte-identical
  	// request — same body, same headers, same signal).
  	const guardedEventStream = emptyCompletionGuardEnabled
  	? guardEmptyCompletionEventStream({
  	firstEvents: eventStream,
  	continuation: fetchGuardedEmptyCompletionRetry,
  	})
  	: eventStream;
  	const { toolNsMap, declaredToolNames, toolParameterSchemas, freeformToolNames, toolSearchToolNames } = toolBridgeMaps;
  	const sseStream = bridgeToResponsesSSE(
  	guardedEventStream, parsed._responseModelId ?? parsed.modelId, toolNsMap, freeformToolNames, toolSearchToolNames,
  	() => upstream.abort(), 2_000,
  	{
  	translatorBudget,
  	replayCacheScope: parsed._reasoningReplayScope,
  	...(options.forceEmptyResponseId ? { responseId: "" } : {}),
  	stallTimeoutSec: config.stallTimeoutSec,
  	hideThinkingSummary: parsed.options.hideThinkingSummary,
  	declaredToolNames,
  	toolParameterSchemas,
  	...(options.onFirstOutput ? { onFirstOutput: options.onFirstOutput } : {}),
  	...(routedCompaction ? { compaction: true } : {}),
  	// Same grok-surface split as the runTurn branch above.
  	...(logCtx.surface === "grok" ? { heartbeatStyle: "comment" as const } : {}),
  	onUsage: usage => {
  	// Raw adapter usage, pre wire-normalization (see the runTurn branch above).
  	logCtx.usageFromBridge = true;

This is distinct from #1572, which tracks outcome-aware failover for routing profiles. These requests already use explicit strategy: "failover" combos, which do advance on ordinary pre-stream 429/5xx failures. It is also distinct from #590, which concerned reset-derived cooldowns blocking later same-account models.

Reproduction

Case A: terminal SSE error before first output

1. Configure a failover combo with at least two eligible OpenAI-compatible targets.
2. Make target A return HTTP 200 with text/event-stream, then a terminal provider error before any text/tool output. The captured provider emitted 服务繁忙，请稍后重试。 (“Service busy, please try again later”).
3. Keep target B healthy and eligible.
4. Send a streaming /v1/responses request through combo/<id> from Codex App.
5. Observe that the logical request ends as 502 and target B is not attempted.

This was observed 11 times in one session. Every affected row had:

• firstOutputMs: null
• an eligible synthetic/hf:zai-org/GLM-5.2 candidate
• a final Openference/GLM-5.2 attempt with status 502
• no subsequent Synthetic attempt

Representative request with Openference selected directly:

2026-08-19T06:59:05Z
requestId: ocx-mszqqnpo-zv
requestedModel: combo/glm-5.2-wrapper
status: 502
firstOutputMs: null
terminalStatus: failed
attempts:
  1. Openference/GLM-5.2 -> 502 upstream_server_error
route candidates:
  nvidia/z-ai/glm-5.2 -> cooldown
  Openference/GLM-5.2 -> selected
  synthetic/hf:zai-org/GLM-5.2 -> eligible, not selected
upstreamError: "Provider stream error: 服务繁忙，请稍后重试。"

Representative request where a pre-stream 429 already advanced correctly, but the later zero-output SSE failure stopped the chain:

2026-08-19T06:53:59Z
requestId: ocx-mszqk3fb-zh
status: 502
firstOutputMs: null
attempts:
  1. nvidia/z-ai/glm-5.2 -> 429
  2. Openference/GLM-5.2 -> 502 upstream_server_error
route candidates also included eligible synthetic/hf:zai-org/GLM-5.2
actual: no third attempt

Case B: model-EOL HTTP 410

1. Configure a failover combo whose first target is nvidia/z-ai/glm-5.2 and whose later targets are healthy/eligible GLM-5.2 providers.
2. Send any request after NVIDIA has removed that model.
3. NVIDIA returns HTTP 410 with a model-EOL message.
4. Observe that OpenCodex returns 410 without trying the remaining targets.

Codex App retried the same turn six times between 04:51:09Z and 04:51:18Z. Every logical request selected NVIDIA and recorded exactly one attempt, despite four eligible candidates.

requestId: ocx-mt5bxjta-2fb
requestedModel: combo/glm-5.2-wrapper
status: 410
firstOutputMs: null
attempts:
  1. nvidia/z-ai/glm-5.2 -> 410 http_410
eligible candidates:
  - nvidia/z-ai/glm-5.2
  - synthetic/hf:zai-org/GLM-5.2
  - Openference/GLM-5.2
  - openrouter/z-ai/glm-5.2:free
upstreamError: "The model 'z-ai/glm-5.2' has reached its end of life on 2026-08-21T09:00:00Z and is no longer available."

The Codex turn then completed with:

unexpected status 410 Gone: Provider error 410: ... model ... reached its end of life ...
url: http://127.0.0.1:10200/v1/responses

Removing the dead NVIDIA target and starting the same prompt again routed to Openference and succeeded. That was a manual/config workaround, not same-request combo failover.

A conservative implementation could preflight/buffer combo child events only until either the first meaningful output is observed, after which no cross-target replay is allowed, or a terminal retryable error is observed before meaningful output, in which case the next target is attempted. For HTTP 410, a structured provider/model lifecycle classification would be safer than making every 4xx retryable.

Version

Observed on @bitkyc08/opencodex 2.29.0 for the HTTP 410 incident. The machine was subsequently updated to 2.31.0; source inspection confirms the same decision/path in 2.31.0 and current dev commit 4f41a8e936141af7ee828e335da314b9dc1ef761.

Operating system

CachyOS, Linux 6.19.11-1-cachyos, x86_64

Provider and model

Failover combo for GLM-5.2 using NVIDIA NIM, Openference, Synthetic, and OpenRouter targets.

Logs or error output

# Zero-output terminal stream failures (11 captured rows)
status=502
errorCode=upstream_server_error
firstOutputMs=null
upstreamError="Provider stream error: 服务繁忙，请稍后重试。"
# A later eligible combo target was not attempted.

# Model EOL (6 consecutive logical requests)
status=410
errorCode=http_410
firstOutputMs=null
attemptCount=1
upstreamError="... z-ai/glm-5.2 ... reached its end of life ... and is no longer available."
# Every route trace listed multiple eligible candidates.

Screenshots and supporting files

No screenshots are required. The request IDs and redacted usage rows above are available from ~/.opencodex/usage.jsonl. Full redacted rows can be attached if requested.

Redacted configuration

{
  "combos": {
    "glm-5.2-wrapper": {
      "strategy": "failover",
      "stickyLimit": 1,
      "defaultEffort": "max",
      "targets": [
        { "provider": "nvidia", "model": "z-ai/glm-5.2", "weight": 1 },
        { "provider": "Openference", "model": "GLM-5.2", "weight": 1 },
        { "provider": "synthetic", "model": "hf:zai-org/GLM-5.2", "weight": 1 },
        { "provider": "openrouter", "model": "z-ai/glm-5.2:free", "weight": 1 }
      ]
    }
  }
}

Checks

• I searched existing issues and documentation.
• I removed secrets, tokens, account details, request credentials, and personal data.

[Ingwannu]

Confirmed on current dev at 4f41a8e93. This is a real combo-boundary bug, and both reported cases are still reachable:

• handleComboResponses commits any response.ok child immediately, before the downstream SSE body is consumed. An OpenAI-compatible HTTP-200 stream can therefore emit a terminal response.failed with no text/tool output after the parent has already exited the failover loop.
• comboFailureDecision does not classify HTTP 410 model-lifecycle failures as target-local, so a dead model stops the chain and remains immediately selectable on the next request.

I am going to keep the fix narrow and fail closed:

1. add a bounded combo-only SSE preflight module that owns one reader and buffers the child response only until either a replay-safety output boundary or a protocol terminal is observed;
2. hop only when a retryable response.failed terminal arrives before any client-visible text, reasoning, tool, or other non-control output; once output begins, preserve the current no-replay behavior;
3. rebuild the failed terminal through the existing bounded combo-failure classifier so status, structured code, usage, and attempt receipts stay consistent;
4. recognize only structured/model-lifecycle HTTP 410 evidence as target-local, cool that exact combo target, and keep unrelated 410 responses terminal;
5. add end-to-end regressions for zero-output SSE failover, post-output failure no-replay, model-EOL 410 failover/cooldown, bounded attempts, and per-attempt logging;
6. document the stream commit boundary and lifecycle cooldown rule in the transport/failover architecture notes.

I will link the implementation PR here after the focused tests, typecheck, documentation build, and full suite pass under the repository's CPU cap. No repository-wide security scan is part of this task.

[Ingwannu]

Implementation is now available in draft PR #2433 at head c742331fa (base dev@4f41a8e93).

The patch adds the bounded, one-reader combo SSE preflight described above and keeps it in a dedicated module. It also adds the narrow model-lifecycle 410 decision, target cooldown, ordered attempt/usage preservation, public combo documentation, and architecture decision log.

Verified on the PR head:

• focused combo/preflight regression suite: 111 passed, 0 failed, 660 expectations;
• TypeScript typecheck: passed;
• docs-site build: passed (393 pages);
• git diff --check: passed.

The CPU-capped full suite did not finish green locally: 14,343 passed, 16 skipped, 10 failed, and 7 unhandled harness errors after 844 seconds. The three named failures are existing tests/codex-shim.test.ts failures and reproduce on an unmodified detached dev@4f41a8e93 worktree. Because the remaining seven harness errors were not attributable from the runner output, #2433 stays draft pending exact-head repository CI and non-author maintainer review.

This issue should remain open until the PR lands. No repository-wide security/privacy scan was run for this task.

[lidge-jun]

리뷰 · 우선순위 66 / 80

설명: 이 이슈는 실패하면 다음으로 넘어가라고 만든 콤보가, 첫 글자가 나오기 전에 죽었는데도 다음 목표를 안 가본다고 한다. 지금 CURRENT dev HEAD 는 4f41a8e93 이다. 이번 시간에 origin/dev 는 그대로다. 새 머지는 없다. 착지는 여전히 2396 사용량 CLI 오늘 비용이다. package.json 은 2.27.0 이다. src/config.ts 는 3975줄이다. src/runtime 폴더는 지금 HEAD 에 없다. 작성자는 2.29.0 에서 410 을 보았고, 2.31.0 과 지금 HEAD 4f41a8e93 에도 같은 길이 있다고 적었다. 내가 지금 HEAD 를 열어 보니 그 두 구멍이 그대로다.

첫 번째 구멍은 이런 일이다. 제공자가 에이치티티피 200 에스에스를 연 다음, 글이나 도구가 나오기 전에 서비스가 바쁩니다 같은 끝 에러를 보낸다. 오픈코덱스는 그 시도를 502 로 기록한다. 그런데 콤보 부모는 자식 응답이 괜찮다고 보고 바로 루프를 나간다. 스트림 본문은 아직 읽기 전이다. 작성자가 한 자리에서 11번 보았다. 매번 firstOutputMs 는 비어 있었다. 옆의 synthetic/hf:zai-org/GLM-5.2 는 쓸 수 있는데도 시도가 없었다. 한 번은 앞 목표가 429 로 넘어간 뒤, 두 번째 목표의 빈 에스에스 실패에서 멈췄다. 세 번째 시도는 없었다.

두 번째 구멍은 모델 수명이 끝난 410 이다. 엔비디아가 z-ai/glm-5.2 를 지워서 410 을 준다. 콤보 실패 판단은 401, 403, 404, 408, 429 와 500대만 다음으로 넘긴다. 410 은 그 목록에 없어서 멈춘다. 죽은 목표를 식히지도 않는다. 코덱스 앱이 같은 턴을 여섯 번 다시 보냈고, 매번 엔비디아만 한 번 고르고 끝났다. 옆의 세 목표는 쓸 수 있었다. 죽은 칸을 설정에서 빼자 그제야 다른 제공자로 가서 성공했다. 그건 같은 요청 안에서의 넘어가기가 아니다.

지금 HEAD 의 src/server/responses/core.ts 1943줄은 자식 응답이 괜찮으면 바로 돌려준다. 에스에스 본문을 미리 보지 않는다. src/combos/failover.ts 147줄 넘어가기 목록에 410 이 없다. 159줄은 나머지 상태를 멈춘다. tests/combos.test.ts 351줄 시험도 401 403 404 408 429 500 503 만 넘긴다고 잠근다. 410 시험은 없다. 빈 에스에스 실패를 다음으로 넘기는 시험도 없다. 식히기 함수는 62줄에 이미 있다. 410 전용 수명 끝 분류는 없다. 메인테이너가 같은 HEAD 에서 두 구멍 모두 닿는다고 확인했다. 고치는 풀 리퀘스트 2433 이 드래프트로 올라왔다. 착지 전에는 이 이슈를 닫지 않는다. 1572 는 라우팅 프로필의 결과 인식 넘어가기다. 이 이슈는 이미 실패하면 다음으로 가라고 적힌 콤보다. 590 은 리셋에서 온 식히기가 같은 계정 다음 모델을 막던 일이다. 둘 다 이 이슈만으로 닫지 않는다. 사용자 길이로는 다음 목표가 있는데도 콤보가 502 나 410 으로 끝나는 구멍이라서 66. 카탈로그 팁은 Ox Alpha x-preview-f-free + deepseek-v4-flash-vision-exp. Cursor 정적 카탈로그는 opus-4-8-fast / opus-5-fast. 2334 CursorCredentialRouter 는 여전히 src/providers/cursor-pool.ts 모듈+테스트만 있고 어댑터에 연결되지 않았다. 2332 H2 는 discovery 전용. 2320 overflow + 2342 는 이미 dev. 2188 사이드카는 이미 dev. 2382 데스크톱 앱 재시작은 이미 dev. 2292 는 아직 연다.

src/server/responses/core.ts 라인 1943 - 자식 응답이 괜찮으면 바로 돌려준다. 에스에스 본문을 미리 보지 않는다
src/server/responses/core.ts 라인 1966 - 콜백 문을 연 채로 루프를 나간다. 그 뒤에 끝 에러가 와도 다음 목표를 안 본다
src/combos/failover.ts 라인 147 - 넘어가기는 401 403 404 408 429 와 500대다. 410 은 없다
src/combos/failover.ts 라인 159 - 나머지 상태는 멈춘다. 수명 끝 410 도 여기로 떨어진다
src/combos/failover.ts 라인 62 - 식히기 함수는 이미 있다. 410 전용 수명 끝 분류는 없다
tests/combos.test.ts 라인 351 - 지금 HEAD 시험은 410 과 빈 에스에스 실패를 잠그지 않는다
PR 2433 - 고치는 드래프트다. 착지 전에는 이 이슈를 닫지 않는다
이슈 1572 - 라우팅 프로필의 다른 길이다. 이 이슈만으로 닫지 않는다

메인테이너의 판단이 필요한 지점

• 출력 전에만 버퍼해서 넘길지. 글이 나온 뒤에는 지금처럼 다시 보내지 않는 편이 맞다
• 410 을 전부 넘길지, 모델 수명이 끝났다는 증거가 있을 때만 넘길지. 관계 없는 410 은 멈추는 편이 맞다
• 메인 2.31.0 핫픽스인지, origin/dev 에 먼저 넣을지. 지금 HEAD 에도 같은 구멍이 있다
• 1572 와 한 이슈로 묶지 말 것. 길이 다르다

너의 추천
이슈는 열어 둔다. 2433 이 착지하면 그 PR 로 닫는다. 첫 글자 전에만 넘긴다. 글이 나온 뒤에는 다시 보내지 말 것. 410 은 모델 수명이 끝난 증거만 다음으로 넘기고 그 목표만 식힌다. 1572 와 2210 과 2279 는 닫지 않는다. 호출 길을 넓히지 말 것. 라벨은 그대로 둔다. 프리뷰 배포가 아니다.

이 댓글은 grok-bot이 작성했습니다

[Ingwannu]

Exact-head repository CI is now green for PR #2433 at c742331faece32575a529a84ca40c974e9a85951 after rerunning the single Linux shard that had failed on an ephemeral EADDRINUSE harness collision. The rerun passed, and every required cross-platform check is successful or intentionally skipped by the workflow. The PR has been moved out of draft and remains assigned to @lidge-jun for non-author maintainer review. I will not self-approve or merge it. This issue remains open until the fix lands on dev.

[Ingwannu]

PR #2433 has been rebased onto current dev and hardened on exact head 83285a6 after review. The follow-up closes the oversized-chunk copy boundary, keeps custom runTurn transports on their existing event-queue preflight, recognizes retirement wording, tests the production classifier directly, and synchronizes the four localized combo guides. Focused result: 113 passed, 0 failed; typecheck and the 393-page docs build pass. Fresh exact-head CI and independent maintainer review are still required, so this issue remains open.

[Ingwannu]

Final exact-head CI completed successfully for PR #2433 at 3ec2b1a after the last rebase onto dev@bf8bcfd3c.

All Linux shards, macOS, storage/API/gates, keyring, npm-global, hygiene, target, CodeRabbit, and aggregate CI checks passed or were intentionally skipped. Local pinned Bun 1.4.0 verification remains 113 passed / 0 failed / 669 expectations, with typecheck, the 393-page docs build, and diff check passing.

The PR checklist now records exact-head CI as green. This issue remains open and the PR remains unmerged until independent maintainer approval.