I understand from the code comments in app/routes/chat.py (around line 500) that LiteLLM can only cascade to a fallback model before the first chunk is emitted:
# NOTE: LiteLLM Router can fall back to a different model only BEFORE the
# first chunk is emitted (or via MidStreamFallbackError, which only some
# providers raise). Once any byte of the SSE stream is sent to the client,
# mid-flight cascade is impossible — we have to surface the error and let
# the client decide what to do.
This is a known limitation, but I'm hitting it in practice with model="auto" + streaming. When the primary model starts streaming successfully but then errors out 20 chunks in (network timeout, upstream 500), the client gets a partial response + an error, and there's no automatic retry on the fallback.
For my use case (long-form content generation), this means ~5% of streaming requests fail partway through, and my client code has to detect the error, re-send the request, and hope the fallback picks up. This defeats the purpose of model="auto"'s reliability promise.
Question: Is there a way to implement a client-side retry wrapper that detects mid-stream failures and re-sends the request with model=<fallback> explicitly? Or would that require the server to expose which fallback to try next via a response header?
I'm open to workarounds, but the current behavior feels like a gap in the "managed safety net" value prop for streaming workloads.
Environment:
- OrcaRouter Lite running on Fly.io (single worker)
- Client: openai-python 1.54.0
- Strategy: balanced
- Primary: claude-opus-4-7, fallback: gpt-5.5
I understand from the code comments in
app/routes/chat.py(around line 500) that LiteLLM can only cascade to a fallback model before the first chunk is emitted:This is a known limitation, but I'm hitting it in practice with
model="auto"+ streaming. When the primary model starts streaming successfully but then errors out 20 chunks in (network timeout, upstream 500), the client gets a partial response + an error, and there's no automatic retry on the fallback.For my use case (long-form content generation), this means ~5% of streaming requests fail partway through, and my client code has to detect the error, re-send the request, and hope the fallback picks up. This defeats the purpose of
model="auto"'s reliability promise.Question: Is there a way to implement a client-side retry wrapper that detects mid-stream failures and re-sends the request with
model=<fallback>explicitly? Or would that require the server to expose which fallback to try next via a response header?I'm open to workarounds, but the current behavior feels like a gap in the "managed safety net" value prop for streaming workloads.
Environment: