From 385e5bc4511722510c2874b8271fa5cde52e3356 Mon Sep 17 00:00:00 2001 From: jiarong0907 Date: Tue, 18 Nov 2025 00:27:06 -0600 Subject: [PATCH 1/3] Add accuracy and cost data --- README.md | 30 +++++++++++++++--------------- 1 file changed, 15 insertions(+), 15 deletions(-) diff --git a/README.md b/README.md index 1cdb628b..12947b12 100644 --- a/README.md +++ b/README.md @@ -32,21 +32,21 @@ RouterArena bridges this gap by providing an open evaluation platform and benchm For more details, please see our [website](https://routeworks.github.io/leaderboard) and [blog](https://huggingface.co/blog/JerryPotter/who-routes-the-routers). -| Rank | Router | Affiliation | Arena | Optimal Selection | Optimal Cost | Optimal Accuracy | Latency | Robustness | -|------|--------------------|-----------------------------|--------|-----------------|--------------|----------------|---------|------------| -| 🥇 | [MIRT‑BERT](https://arxiv.org/pdf/2506.01048) [[GH]](https://github.com/Mercidaiha/IRT-Router) | 🎓 USTC | 66.89 | 3.44 | 19.62 | 78.18 | 27.03 | 94.50 | -| 🥈 | [Azure‑Router](https://ai.azure.com/catalog/models/model-router) [[Web]](https://learn.microsoft.com/en-us/azure/ai-foundry/openai/concepts/model-router) | 💼 Microsoft | 66.66 | 22.52 | 46.32 | 81.96 | — | — | -| 🥉 | [NIRT‑BERT](https://arxiv.org/pdf/2506.01048) [[GH]](https://github.com/Mercidaiha/IRT-Router) | 🎓 USTC | 66.12 | 3.83 | 14.04 | 77.88 | 10.42 | 44.50 | -| 4 | [GPT‑5](https://openai.com/index/introducing-gpt-5/)| 💼 OpenAI | 64.32 | — | — | — | — | — | -| 5 | [vLLM‑SR](https://vllm-semantic-router.com/) [[GH]](https://github.com/vllm-project/semantic-router) [[HF]](https://huggingface.co/llm-semantic-router) | 🎓 vLLM SR Team | 64.32 | 4.79 | 12.54 | 79.33 | 0.19 | 100.00 | -| 6 | [CARROT](https://arxiv.org/abs/2502.03261) [[GH]](https://github.com/somerstep/CARROT) [[HF]](https://huggingface.co/CARROT-LLM-Routing) | 🎓 UMich | 63.87 | 2.68 | 6.77 | 78.63 | 1.50 | 93.60 | -| 7 | [Chayan](https://huggingface.co/adaptive-classifier/chayan) [[HF]](https://huggingface.co/adaptive-classifier/chayan) | 🎓 Adaptive Classifier | 63.83 | 43.03 | 43.75 | 88.74 | — | — | -| 8 | [NotDiamond](https://www.notdiamond.ai/) | 💼 NotDiamond | 63.00 | 1.55 | 2.14 | 76.81 | — | — | -| 9 | [RouterBench‑MLP](https://arxiv.org/pdf/2403.12031) [[GH]](https://github.com/withmartian/routerbench) [[HF]](https://huggingface.co/datasets/withmartian/routerbench) | 🎓 Martian | 57.56 | 13.39 | 24.45 | 83.32 | 90.91 | 96.90 | -| 10 | [GraphRouter](https://arxiv.org/abs/2410.03834) [[GH]](https://github.com/ulab-uiuc/GraphRouter) | 🎓 UIUC | 57.22 | 4.73 | 38.33 | 74.25 | 2.70 | 97.50 | -| 11 | [RouterBench‑KNN](https://arxiv.org/pdf/2403.12031) [[GH]](https://github.com/withmartian/routerbench) [[HF]](https://huggingface.co/datasets/withmartian/routerbench) | 🎓 Martian | 55.48 | 13.09 | 25.49 | 78.77 | 1.33 | 51.30 | -| 12 | [RouteLLM](https://arxiv.org/abs/2406.18665) [[GH]](https://github.com/lm-sys/RouteLLM) [[HF]](https://huggingface.co/routellm) | 🎓 Berkeley | 48.07 | 99.72 | 99.63 | 68.76 | 0.40 | 99.80 | -| 13 | [RouterDC](https://arxiv.org/abs/2409.19886) [[GH]](https://github.com/shuhao02/RouterDC) | 🎓 SUSTech | 33.75 | 39.84 | 73.00 | 49.05 | 10.75 | 97.60 | +| Rank | Router | Affiliation | Acc-Cost Arena | Accuracy | Cost/1K | Optimal Selection | Optimal Cost | Optimal Accuracy | Latency | Robustness | +|------|--------------------|-----------------------------|--------|----------|---------|-----------------|--------------|----------------|---------|------------| +| 🥇 | [MIRT‑BERT](https://arxiv.org/pdf/2506.01048) [[GH]](https://github.com/Mercidaiha/IRT-Router) | 🎓 USTC | 66.89 | 60.88 | $0.15 | 3.44 | 19.62 | 78.18 | 27.03 | 94.50 | +| 🥈 | [Azure‑Router](https://ai.azure.com/catalog/models/model-router) [[Web]](https://learn.microsoft.com/en-us/azure/ai-foundry/openai/concepts/model-router) | 💼 Microsoft | 66.66 | 68.09 | $0.54 | 22.52 | 46.32 | 81.96 | — | — | +| 🥉 | [NIRT‑BERT](https://arxiv.org/pdf/2506.01048) [[GH]](https://github.com/Mercidaiha/IRT-Router) | 🎓 USTC | 66.12 | 66.34 | $0.21 | 3.83 | 14.04 | 77.88 | 10.42 | 44.50 | +| 4 | [GPT‑5](https://openai.com/index/introducing-gpt-5/)| 💼 OpenAI | 64.32 | 73.96 | $10.02 | — | — | — | — | — | +| 5 | [vLLM‑SR](https://vllm-semantic-router.com/) [[GH]](https://github.com/vllm-project/semantic-router) [[HF]](https://huggingface.co/llm-semantic-router) | 🎓 vLLM SR Team | 64.32 | 67.28 | $1.67 | 4.79 | 12.54 | 79.33 | 0.19 | 100.00 | +| 6 | [CARROT](https://arxiv.org/abs/2502.03261) [[GH]](https://github.com/somerstep/CARROT) [[HF]](https://huggingface.co/CARROT-LLM-Routing) | 🎓 UMich | 63.87 | 67.21 | $2.06 | 2.68 | 6.77 | 78.63 | 1.50 | 93.60 | +| 7 | [Chayan](https://huggingface.co/adaptive-classifier/chayan) [[HF]](https://huggingface.co/adaptive-classifier/chayan) | 🎓 Adaptive Classifier | 63.83 | — | — | 43.03 | 43.75 | 88.74 | — | — | +| 8 | [NotDiamond](https://www.notdiamond.ai/) | 💼 NotDiamond | 63.00 | 60.83 | $4.10 | 1.55 | 2.14 | 76.81 | — | — | +| 9 | [RouterBench‑MLP](https://arxiv.org/pdf/2403.12031) [[GH]](https://github.com/withmartian/routerbench) [[HF]](https://huggingface.co/datasets/withmartian/routerbench) | 🎓 Martian | 57.56 | 61.62 | $4.83 | 13.39 | 24.45 | 83.32 | 90.91 | 96.90 | +| 10 | [GraphRouter](https://arxiv.org/abs/2410.03834) [[GH]](https://github.com/ulab-uiuc/GraphRouter) | 🎓 UIUC | 57.22 | 57.00 | $0.34 | 4.73 | 38.33 | 74.25 | 2.70 | 97.50 | +| 11 | [RouterBench‑KNN](https://arxiv.org/pdf/2403.12031) [[GH]](https://github.com/withmartian/routerbench) [[HF]](https://huggingface.co/datasets/withmartian/routerbench) | 🎓 Martian | 55.48 | 58.69 | $4.27 | 13.09 | 25.49 | 78.77 | 1.33 | 51.30 | +| 12 | [RouteLLM](https://arxiv.org/abs/2406.18665) [[GH]](https://github.com/lm-sys/RouteLLM) [[HF]](https://huggingface.co/routellm) | 🎓 Berkeley | 48.07 | 47.04 | $0.27 | 99.72 | 99.63 | 68.76 | 0.40 | 99.80 | +| 13 | [RouterDC](https://arxiv.org/abs/2409.19886) [[GH]](https://github.com/shuhao02/RouterDC) | 🎓 SUSTech | 33.75 | 32.01 | $0.07 | 39.84 | 73.00 | 49.05 | 10.75 | 97.60 | 🎓 Open-source  💼 Closed-source  From 29afac3334012beac3fcb4a4774795f8e51af7eb Mon Sep 17 00:00:00 2001 From: jiarong0907 Date: Tue, 18 Nov 2025 00:29:47 -0600 Subject: [PATCH 2/3] Data for Chayan --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 12947b12..bc8db3e7 100644 --- a/README.md +++ b/README.md @@ -32,7 +32,7 @@ RouterArena bridges this gap by providing an open evaluation platform and benchm For more details, please see our [website](https://routeworks.github.io/leaderboard) and [blog](https://huggingface.co/blog/JerryPotter/who-routes-the-routers). -| Rank | Router | Affiliation | Acc-Cost Arena | Accuracy | Cost/1K | Optimal Selection | Optimal Cost | Optimal Accuracy | Latency | Robustness | +| Rank | Router | Affiliation | Acc-Cost Arena | Accuracy | Cost/1K Queries | Optimal Selection | Optimal Cost | Optimal Accuracy | Latency | Robustness | |------|--------------------|-----------------------------|--------|----------|---------|-----------------|--------------|----------------|---------|------------| | 🥇 | [MIRT‑BERT](https://arxiv.org/pdf/2506.01048) [[GH]](https://github.com/Mercidaiha/IRT-Router) | 🎓 USTC | 66.89 | 60.88 | $0.15 | 3.44 | 19.62 | 78.18 | 27.03 | 94.50 | | 🥈 | [Azure‑Router](https://ai.azure.com/catalog/models/model-router) [[Web]](https://learn.microsoft.com/en-us/azure/ai-foundry/openai/concepts/model-router) | 💼 Microsoft | 66.66 | 68.09 | $0.54 | 22.52 | 46.32 | 81.96 | — | — | @@ -40,7 +40,7 @@ For more details, please see our [website](https://routeworks.github.io/leaderbo | 4 | [GPT‑5](https://openai.com/index/introducing-gpt-5/)| 💼 OpenAI | 64.32 | 73.96 | $10.02 | — | — | — | — | — | | 5 | [vLLM‑SR](https://vllm-semantic-router.com/) [[GH]](https://github.com/vllm-project/semantic-router) [[HF]](https://huggingface.co/llm-semantic-router) | 🎓 vLLM SR Team | 64.32 | 67.28 | $1.67 | 4.79 | 12.54 | 79.33 | 0.19 | 100.00 | | 6 | [CARROT](https://arxiv.org/abs/2502.03261) [[GH]](https://github.com/somerstep/CARROT) [[HF]](https://huggingface.co/CARROT-LLM-Routing) | 🎓 UMich | 63.87 | 67.21 | $2.06 | 2.68 | 6.77 | 78.63 | 1.50 | 93.60 | -| 7 | [Chayan](https://huggingface.co/adaptive-classifier/chayan) [[HF]](https://huggingface.co/adaptive-classifier/chayan) | 🎓 Adaptive Classifier | 63.83 | — | — | 43.03 | 43.75 | 88.74 | — | — | +| 7 | [Chayan](https://huggingface.co/adaptive-classifier/chayan) [[HF]](https://huggingface.co/adaptive-classifier/chayan) | 🎓 Adaptive Classifier | 63.83 | 64.89 | $0.56 | 43.03 | 43.75 | 88.74 | — | — | | 8 | [NotDiamond](https://www.notdiamond.ai/) | 💼 NotDiamond | 63.00 | 60.83 | $4.10 | 1.55 | 2.14 | 76.81 | — | — | | 9 | [RouterBench‑MLP](https://arxiv.org/pdf/2403.12031) [[GH]](https://github.com/withmartian/routerbench) [[HF]](https://huggingface.co/datasets/withmartian/routerbench) | 🎓 Martian | 57.56 | 61.62 | $4.83 | 13.39 | 24.45 | 83.32 | 90.91 | 96.90 | | 10 | [GraphRouter](https://arxiv.org/abs/2410.03834) [[GH]](https://github.com/ulab-uiuc/GraphRouter) | 🎓 UIUC | 57.22 | 57.00 | $0.34 | 4.73 | 38.33 | 74.25 | 2.70 | 97.50 | From c4dbcc6086b3a94c2faf009c20e8ee061c0b0ff5 Mon Sep 17 00:00:00 2001 From: jiarong0907 Date: Tue, 18 Nov 2025 00:33:31 -0600 Subject: [PATCH 3/3] fix error --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index bc8db3e7..c7d52380 100644 --- a/README.md +++ b/README.md @@ -34,7 +34,7 @@ For more details, please see our [website](https://routeworks.github.io/leaderbo | Rank | Router | Affiliation | Acc-Cost Arena | Accuracy | Cost/1K Queries | Optimal Selection | Optimal Cost | Optimal Accuracy | Latency | Robustness | |------|--------------------|-----------------------------|--------|----------|---------|-----------------|--------------|----------------|---------|------------| -| 🥇 | [MIRT‑BERT](https://arxiv.org/pdf/2506.01048) [[GH]](https://github.com/Mercidaiha/IRT-Router) | 🎓 USTC | 66.89 | 60.88 | $0.15 | 3.44 | 19.62 | 78.18 | 27.03 | 94.50 | +| 🥇 | [MIRT‑BERT](https://arxiv.org/pdf/2506.01048) [[GH]](https://github.com/Mercidaiha/IRT-Router) | 🎓 USTC | 66.89 | 66.88 | $0.15 | 3.44 | 19.62 | 78.18 | 27.03 | 94.50 | | 🥈 | [Azure‑Router](https://ai.azure.com/catalog/models/model-router) [[Web]](https://learn.microsoft.com/en-us/azure/ai-foundry/openai/concepts/model-router) | 💼 Microsoft | 66.66 | 68.09 | $0.54 | 22.52 | 46.32 | 81.96 | — | — | | 🥉 | [NIRT‑BERT](https://arxiv.org/pdf/2506.01048) [[GH]](https://github.com/Mercidaiha/IRT-Router) | 🎓 USTC | 66.12 | 66.34 | $0.21 | 3.83 | 14.04 | 77.88 | 10.42 | 44.50 | | 4 | [GPT‑5](https://openai.com/index/introducing-gpt-5/)| 💼 OpenAI | 64.32 | 73.96 | $10.02 | — | — | — | — | — |