Skip to content

model="auto" with preferred_models silently ignores non-deployable entries instead of erroring #109

Description

@najwac

Description

When I configure preferred_models via PUT /v1/routing with model IDs that don't have a configured provider key, the router silently falls back to the full eligible set instead of surfacing the misconfiguration. This makes it really hard to debug why my preferred tier isn't being honored.

Steps to Reproduce

  1. Start Lite with only OPENAI_API_KEY set (no Anthropic key)
  2. PUT /v1/routing with {"strategy": "quality", "preferred_models": ["claude-opus-4-7", "gpt-5.5"]}
  3. Send POST /v1/chat/completions with model="auto"

Expected vs Actual

Expected: Either a 422 saying "preferred model claude-opus-4-7 has no deployable provider" or at minimum an x-orca-warning header indicating the preferred set was empty after filtering.

Actual: The request succeeds, routes to the cheapest available OpenAI model, and the response looks identical to a cheapest strategy call. The dashboard shows the workspace strategy as "quality" which is misleading.

Environment

  • OrcaRouter Lite main (commit ac46b84)
  • Docker on Ubuntu 24.04
  • Python 3.12.7

The narrowing logic in app/auto_routing.py:choose_auto_model (around line 230) does:

if preferred_models:
    preferred_set = set(preferred_models)
    narrowed = [m for m in eligible if m.id in preferred_set]
    if narrowed:
        eligible = narrowed

The if narrowed guard means when the intersection is empty, the full eligible set passes through silently. I think this should at least log a warning, or better yet, return metadata that the caller can surface.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions