Update Azure Model Router Evaluation

[codebyshibsankar]

Azure Model Router (2025-08-12) — submission

Affiliation: Microsoft — Azure Model Router

Azure's built-in model router, available through the Azure AI Foundry platform. It automatically routes prompts across a pool of 27 models spanning 5 providers, all accessible through a single Azure OpenAI endpoint.

How it routes

As a trained language model, the router analyzes prompts in real time based on complexity, reasoning, task type, and other attributes. It does not store prompts and routes only to eligible models based on access and deployment types, honoring data-zone boundaries.

Supported Models

• OpenAI: gpt-4.1-nano, gpt-5-nano, gpt-4o-mini, gpt-oss-120b, gpt-5.4-nano, gpt-4.1-mini, gpt-4.1, gpt-5-mini, gpt-5.4-mini, o4-mini, gpt-5.2, gpt-4o, gpt-5, gpt-5.4, gpt-5.5, gpt-5.6-luna, gpt-5.6-terra, gpt-5.6-sol
• Anthropic: claude-haiku-4-5, claude-sonnet-4-5, claude-opus-4-6, claude-opus-4-7, claude-opus-4.8
• DeepSeek: deepseek-v3.2
• xAI: grok-4, grok-4-1-fast-reasoning
• Meta: llama4-maverick-instruct

Routing Mode

With the latest version, if you choose a custom deployment, you can select a routing mode to optimize for quality or cost while maintaining a baseline level of performance. Setting a routing mode is optional; if you do not set one, your deployment defaults to Balanced mode.

Available routing modes:

• Balanced: Considers both cost and quality dynamically; ideal for general-purpose scenarios.
• Quality: Prioritizes maximum accuracy; best for complex reasoning or critical outputs.
• Cost: Prioritizes cost savings; ideal for budget-sensitive or high-volume workloads.

Model Subset

You can specify which underlying models are included in routing decisions. This gives more control over cost, compliance, and performance characteristics.

Automatic Failover

Model router includes built-in automatic failover. When using the default deployment to route to all supported models, it transparently redirects requests to the next-most-appropriate model so transient issues with any single model do not disrupt the application. Failover is enabled by default.

Prompt Caching

Model router supports prompt caching because requests are processed by the underlying models that support it. When the router delegates a request to a model that supports prompt caching, cached tokens are used automatically with no extra configuration.

Files Changed

• azure-model-router.json
• azure-model-router-may-26.json — 8,400 regular predictions
• azure-model-router-robustness-may-26.json — 420 robustness routes
• universal_model_names.py — additive updates for newer model aliases and mappings
• model_cost.json — additive pricing entries for newer models

This evaluation was done in cost mode with the full 27-model routing set.

[codebyshibsankar]

/evaluate

[github-actions]

Router Evaluation Results

Router: azure-model-router
Dataset Split: full

RouterArena Metrics

Metric	Value
RouterArena Score	0.7141
Accuracy	74.23%
Total Cost	$6.538387
Avg Cost per Query	$0.000778
Avg Cost per 1K Queries	$0.7784
Number of Queries	8400
Abnormal Entries	0
Robustness Score	0.8714

---

Evaluation completed by RouterArena automated workflow