Skip to content

Revert PR 20 #21

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jiarong0907 merged 1 commit into from
Nov 11, 2025
Merged

Revert PR 20 #21

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
98 changes: 52 additions & 46 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,56 +11,54 @@

</div>

<h1 align="center"> Make Router Evaluation Open and Standardized </h1>
<h2 align="center"> Make Router Evaluation Open and Standardized </h2>

<p align="center">
<img src="images/routerarena-diagram.png" alt="RouterArena Diagram" width="700" />
<img src="images/routerarena-diagram.png" alt="RouterArena Diagram" width="1000" />
</p>

**RouterArena** is an open evaluation platform and leaderboard for **LLM routers**—systems that automatically select the best model for a given query. As the LLM ecosystem diversifies with models varying in size, capability, and cost, **routing** has become critical for balancing performance and cost. Yet, LLM routers currently lack a standardized evaluation framework to assess how effectively they trade off accuracy, cost, and other related metrics.
**RouterArena** is an open evaluation platform and leaderboard for **LLM routers** — systems that automatically select the best model for a given query. As the LLM ecosystem diversifies into specialized models of varying size, capability, and cost, routing has become essential for balancing performance and efficiency. Yet, unlike models, routers currently lack a unified evaluation standard that measures how well they trade off accuracy, cost, robustness, and latency.

RouterArena bridges this gap by providing an open evaluation platform and benchmarking framework for both open-source and commercial routers. It has the following key features:
RouterArena addresses this gap by providing a uniform, multi-dimensional benchmarking framework for both open-source and commercial routers. It introduces a principled dataset with diverse domains and difficulty levels, a comprehensive suite of evaluation metrics, and an automated leaderboard for transparent comparison. By standardizing router evaluation, RouterArena lays the foundation for reproducible, fair, and continuous progress in the next generation of routing systems.

- 🌍 **Diverse Data Coverage**: A principly-constructed, diverse evaluation dataset spanning 9 domains and 44 categories with easy, medium, and hard difficulty levels.
- 📊 **Comprehensive Metrics**: Five router-critical metrics measuring accuracy, cost, optimality, robustness, and latency.
- ⚙️ **Automated Evaluation**: An automated evaluation framework to simplify the evaluation process for open-source and commercial routers.
- 🏆 **Live Leaderboard**: A live leaderboard to track the performance of routers across multiple dimensions.
<h3 align="left">Key Features</h3>

*We aim for RouterArena to serve as a foundation for the community to evaluate, understand, and advance LLM routing systems.*
- **Diverse Data Coverage**: A principled, diverse evaluation dataset spanning 9 domains and 44 categories
- **Comprehensive Eval Metrics**: Five complementary evaluation metrics capturing accuracy, cost, optimality, robustness, and latency.
- **Uniform Eval Framework**: Fairly benchmarked open-sourced and commercial routers.
- **Live Leaderboard**: Ranking routers across multiple dimensions.

# Current Leaderboard

For more details, please see our [website](https://routeworks.github.io/leaderboard) and [blog](https://huggingface.co/blog/JerryPotter/who-routes-the-routers).
<h3 align="left">RouterArena Leaderboard</h3>

| Rank | Router | Affiliation | Arena | Opt.Sel | Opt.Cost | Opt.Acc | Lat | Robust |
|------|---------|--------------|--------|------------|-----------|----------|----------|-------------|
| 🥇 | MIRT-BERT | 🎓 USTC | 66.89 | 3.44 | 19.62 | 78.18 | 27.03 | 94.50 |
| 🥈 | Azure | 💼 Microsoft | 66.66 | 22.52 | 46.32 | 81.96 | — | — |
| 🥉 | NIRT-BERT | 🎓 USTC | 66.12 | 3.83 | 14.04 | 77.88 | 10.42 | 44.50 |
| 4 | GPT-5 | 💼 OpenAI | 64.32 | — | — | — | — | — |
| 5 | vLLM-SR | 💼 vLLM | 64.32 | 4.79 | 12.54 | 79.33 | 0.19 | 100.00 |
| 6 | CARROT | 🎓 UMich | 63.87 | 2.68 | 6.77 | 78.63 | 1.50 | 93.60 |
| 7 | NotDiamond | 💼 NotDiamond | 63.00 | 1.55 | 2.14 | 76.81 | — | — |
| 8 | MLP | 🎓 Academic | 57.56 | 13.39 | 24.45 | 83.32 | 90.91 | 96.90 |
| 9 | GraphRouter | 🎓 UIUC | 57.22 | 4.73 | 38.33 | 74.25 | 2.70 | 97.50 |
| 10 | KNN | 🎓 Academic | 55.48 | 13.09 | 25.49 | 78.77 | 1.33 | 51.30 |
| 11 | RouteLLM | 🎓 Berkeley | 48.07 | 99.72 | 99.63 | 68.76 | 0.40 | 99.80 |
| 12 | RouterDC | 🎓 SUSTech | 33.75 | 39.84 | 73.00 | 49.05 | 10.75 | 97.60 |

🎓 Academic  💼 Commercial 
| 🥇 | MIRT-BERT | 🟨 USTC | 66.89 | 3.44 | 19.62 | 78.18 | 27.03 | 94.50 |
| 🥈 | Azure | 🟦&nbsp;Microsoft | 66.66 | 22.52 | 46.32 | 81.96 | — | — |
| 🥉 | NIRT-BERT | 🟨 USTC | 66.12 | 3.83 | 14.04 | 77.88 | 10.42 | 44.50 |
| 4 | GPT-5 | 🟦 OpenAI | 64.32 | — | — | — | — | — |
| 5 | vLLM-SR | 🟦 vLLM | 64.32 | 4.79 | 12.54 | 79.33 | 0.19 | 100.00 |
| 6 | CARROT | 🟨 UMich | 63.87 | 2.68 | 6.77 | 78.63 | 1.50 | 93.60 |
| 7 | NotDiamond | 🟦&nbsp;NotDiamond | 63.00 | 1.55 | 2.14 | 76.81 | — | — |
| 8 | MLP | 🟨 Academic | 57.56 | 13.39 | 24.45 | 83.32 | 90.91 | 96.90 |
| 9 | GraphRouter | 🟨 UIUC | 57.22 | 4.73 | 38.33 | 74.25 | 2.70 | 97.50 |
| 10 | KNN | 🟨 Academic | 55.48 | 13.09 | 25.49 | 78.77 | 1.33 | 51.30 |
| 11 | RouteLLM | 🟨 Berkeley | 48.07 | 99.72 | 99.63 | 68.76 | 0.40 | 99.80 |
| 12 | RouterDC | 🟨 SUSTech | 33.75 | 39.84 | 73.00 | 49.05 | 10.75 | 97.60 |

🟨 Academic  🟦 Commercial 

<!-- <p align="center">
<img src="images/leaderboard.png" alt="Make GPU Sharing Flexible and Easy" width="500" />
</p> -->

<!-- # Have your router on the leaderboard! -->
The current leaderboard is computed considering the accuracy and overall cost for each router. For more details, please read our [blog](https://huggingface.co/blog/JerryPotter/who-routes-the-routers).

# Evaluating Your Router
<h2 align="left">Have your router on there!</h3>

To use our framework to evaluate your router and get your router on the leaderboard, you can follow the steps below. The evaluation pipelines include two stages as shown in the diagram below. First, you need to generate a prediction file for your router. Then, you can open a Pull Request with your router's prediction file to trigger our automated evaluation workflow.
If you want your router on the leaderboard, submit a Pull Request with your router's prediction file. For questions or issues, please open a GitHub issue. For fairness, we have withheld the ground truth answers for the full dataset. However, you can still test your router using the sub-sampled 10% dataset by following the steps below.

<p align="center">
<img src="images/pipeline.png" alt="RouterArena Evaluation Pipeline" width="700" />
<img src="images/pipeline.png" alt="RouterArena Evaluation Pipeline" width="1000" />
</p>

## Setup
Expand All @@ -74,32 +72,34 @@ uv sync
```

### Step 2: Download Dataset
Download the dataset from [HF dataset](https://huggingface.co/datasets/RouteWorks/RouterArena).
Run this command to download the dataset from the [HF dataset](https://huggingface.co/datasets/RouteWorks/RouterArena).

```bash
uv run python ./scripts/process_datasets/prep_datasets.py
```

### Step 3: Set Up API Keys (Optional)
### Step 3: Set Up API Keys

Update the API keys in the `.env` file in the project root. This step is **required only if you use our pipeline for LLM inferences**.
This step is **required only if you plan to use our pipeline to make LLM inferences**. Create a `.env` file in the project root and add the API keys for the providers you need:

```bash
# Example .env file
OPENAI_API_KEY=<Your-Key>
ANTHROPIC_API_KEY=<Your-Key>
HF_TOKEN=<Your-Key>
# ...
```

See the [`ModelInference`](./llm_inference/model_inference.py) class for the complete list of supported providers and required environment variables. You can extend that class to support more models, or submit a GitHub issue to request support for new providers.
#### Optional:
See the `ModelInference` class in `RouterArena/llm_inference/model_inference.py` for the complete list of supported providers and required environment variables. You can extend that class to support additional models, or submit a GitHub issue to request support for new providers.

## Get Routing Decisions
## Usage

Follow the steps below to obtain your router's model choices for each query. Start with the `sub_10` split (a 10% subset with ground-truth answers) for local testing. Once your setup works, you can evaluate on the `full` dataset for official leaderboard submission.
Follow the steps below to evaluate your router. You can start with the `sub_10` split (10% sub-sampled dataset) to test your setup and code. The `sub_10` split includes ground truth answers for local testing. Once ready, you can evaluate on the `full` dataset for official leaderboard submission.

### Step 1: Prepare Config File

Create a config file in `./router_inference/config/<router_name>.json`. An example config file is included [here](./router_inference/config/your-router.json).
Create a config file in `./router_inference/config/<router_name>.json`. We have created an example router for demonstration purposes:

```json
{
Expand All @@ -108,13 +108,16 @@ Create a config file in `./router_inference/config/<router_name>.json`. An examp
"models": [
"gpt-4o-mini",
"claude-3-haiku-20240307",
"gemini-2.0-flash-001"
"gemini-2.0-flash-001",
"mistral-medium"
]
}
}
```

For each model in your config, add an entry with the pricing per million tokens in this format at [`model_cost/cost.json`](./model_cost/cost.json):
*Note: The model name must be the same as the one used in `./universal_model_names.py` (see next step for details)*

**Important**: For each model in your config, add an entry with the pricing per million tokens in this format at `model_cost/cost.json`:

```json
{
Expand All @@ -125,23 +128,26 @@ For each model in your config, add an entry with the pricing per million tokens
}
```

> [!NOTE]
> Ensure all models in your above config files are listed in [`./universal_model_names.py`](./universal_model_names.py). If you add a new model, you must also add the API inference endpoint in [`llm_inference/model_inference.py`](./llm_inference/model_inference.py).
### Step 2: Verify Model Names

### Step 2: Generate Router's Prediction File
Ensure all models in your config are listed in `./universal_model_names.py`. If you add a new model, you must also add the API inference endpoint in `RouterArena/llm_inference/model_inference.py`.

### Step 3: Generate Router's Prediction File

Generate a template prediction file:

```bash
uv run python ./router_inference/generate_prediction_file.py your-router [sub_10|full]
uv run python ./router_inference/generate_prediction_file.py your-router sub_10
```

**Important**: Replace the placeholder model choices in the `prediction` field with your router's actual selections.
Use `full` instead of `sub_10` for the complete dataset. **Important**: Replace the placeholder model choices in the `prediction` field with your router's actual selections.

### Step 4: Validate Config and Prediction Files

### Step 3: Validate Config and Prediction Files
Validate your config and prediction files before proceeding:

```bash
uv run python ./router_inference/check_config_prediction_files.py your-router [sub_10|full]
uv run python ./router_inference/check_config_prediction_files.py your-router sub_10
```

This script checks: (1) all model names are valid, (2) prediction file has correct size (809 for `sub_10`, 8400 for `full`), and (3) all entries have valid `global_index`, `prompt`, and `prediction` fields.
Expand Down
Binary file modified images/pipeline.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.