From 602d8b019499d9c09bd83f4847a539ec6dda58f6 Mon Sep 17 00:00:00 2001 From: jiarong0907 Date: Mon, 17 Nov 2025 21:36:53 -0600 Subject: [PATCH 01/10] Add citations for each router --- README.md | 26 +++++++++++++------------- 1 file changed, 13 insertions(+), 13 deletions(-) diff --git a/README.md b/README.md index d7a750a6..050cd6f2 100644 --- a/README.md +++ b/README.md @@ -34,19 +34,19 @@ For more details, please see our [website](https://routeworks.github.io/leaderbo | Rank | Router | Affiliation | Arena | Opt.Sel | Opt.Cost | Opt.Acc | Lat | Robust | |------|---------|--------------|--------|------------|-----------|----------|----------|-------------| -| πŸ₯‡ | MIRT-BERT | πŸŽ“ USTC | 66.89 | 3.44 | 19.62 | 78.18 | 27.03 | 94.50 | -| πŸ₯ˆ | Azure | πŸ’Ό Microsoft | 66.66 | 22.52 | 46.32 | 81.96 | β€” | β€” | -| πŸ₯‰ | NIRT-BERT | πŸŽ“ USTC | 66.12 | 3.83 | 14.04 | 77.88 | 10.42 | 44.50 | -| 4 | GPT-5 | πŸ’Ό OpenAI | 64.32 | β€” | β€” | β€” | β€” | β€” | -| 5 | vLLM-SR | πŸ’Ό vLLM | 64.32 | 4.79 | 12.54 | 79.33 | 0.19 | 100.00 | -| 6 | CARROT | πŸŽ“ UMich | 63.87 | 2.68 | 6.77 | 78.63 | 1.50 | 93.60 | -| 7 | Chayan | πŸ’Ό Adaptive Classifier | 63.83 | 43.03 | 43.75 | 88.74 | β€” | β€” | -| 8 | NotDiamond | πŸ’Ό NotDiamond | 63.00 | 1.55 | 2.14 | 76.81 | β€” | β€” | -| 9 | MLP | πŸŽ“ Academic | 57.56 | 13.39 | 24.45 | 83.32 | 90.91 | 96.90 | -| 10 | GraphRouter | πŸŽ“ UIUC | 57.22 | 4.73 | 38.33 | 74.25 | 2.70 | 97.50 | -| 11 | KNN | πŸŽ“ Academic | 55.48 | 13.09 | 25.49 | 78.77 | 1.33 | 51.30 | -| 12 | RouteLLM | πŸŽ“ Berkeley | 48.07 | 99.72 | 99.63 | 68.76 | 0.40 | 99.80 | -| 13 | RouterDC | πŸŽ“ SUSTech | 33.75 | 39.84 | 73.00 | 49.05 | 10.75 | 97.60 | +| πŸ₯‡ | [MIRT-BERT](https://arxiv.org/pdf/2506.01048) [[GH]](https://github.com/Mercidaiha/IRT-Router) | πŸŽ“ USTC | 66.89 | 3.44 | 19.62 | 78.18 | 27.03 | 94.50 | +| πŸ₯ˆ | [Azure-Router](https://ai.azure.com/catalog/models/model-router) [[Web]](https://learn.microsoft.com/en-us/azure/ai-foundry/openai/concepts/model-router) | πŸ’Ό Microsoft | 66.66 | 22.52 | 46.32 | 81.96 | β€” | β€” | +| πŸ₯‰ | [NIRT-BERT](https://arxiv.org/pdf/2506.01048) [[GH]](https://github.com/Mercidaiha/IRT-Router) | πŸŽ“ USTC | 66.12 | 3.83 | 14.04 | 77.88 | 10.42 | 44.50 | +| 4 | [GPT-5](https://openai.com/index/introducing-gpt-5/)| πŸ’Ό OpenAI | 64.32 | β€” | β€” | β€” | β€” | β€” | +| 5 | [vLLM-SR](https://vllm-semantic-router.com/) [[GH]](https://github.com/vllm-project/semantic-router) [[HF]](https://huggingface.co/llm-semantic-router) | πŸ’Ό vLLM | 64.32 | 4.79 | 12.54 | 79.33 | 0.19 | 100.00 | +| 6 | [CARROT](https://arxiv.org/abs/2502.03261) [[GH]](https://github.com/somerstep/CARROT) [[HF]](https://huggingface.co/CARROT-LLM-Routing) | πŸŽ“ UMich | 63.87 | 2.68 | 6.77 | 78.63 | 1.50 | 93.60 | +| 7 | [Chayan](https://huggingface.co/adaptive-classifier/chayan) [[HF]](https://huggingface.co/adaptive-classifier/chayan) | πŸ’Ό Adaptive Classifier | 63.83 | 43.03 | 43.75 | 88.74 | β€” | β€” | +| 8 | [NotDiamond](https://www.notdiamond.ai/) | πŸ’Ό NotDiamond | 63.00 | 1.55 | 2.14 | 76.81 | β€” | β€” | +| 9 | [RouterBench-MLP](https://arxiv.org/pdf/2403.12031) [[GH]](https://github.com/withmartian/routerbench) [[HF]](https://huggingface.co/datasets/withmartian/routerbench) | πŸŽ“ Academic | 57.56 | 13.39 | 24.45 | 83.32 | 90.91 | 96.90 | +| 10 | [GraphRouter](https://arxiv.org/abs/2410.03834) [[GH]](https://github.com/ulab-uiuc/GraphRouter) | πŸŽ“ UIUC | 57.22 | 4.73 | 38.33 | 74.25 | 2.70 | 97.50 | +| 11 | [RouterBench-KNN](https://arxiv.org/pdf/2403.12031) [[GH]](https://github.com/withmartian/routerbench) [[HF]](https://huggingface.co/datasets/withmartian/routerbench) | πŸŽ“ Academic | 55.48 | 13.09 | 25.49 | 78.77 | 1.33 | 51.30 | +| 12 | [RouteLLM](https://arxiv.org/abs/2406.18665) [[GH]](https://github.com/lm-sys/RouteLLM) [[HF]](https://huggingface.co/routellm) | πŸŽ“ Berkeley | 48.07 | 99.72 | 99.63 | 68.76 | 0.40 | 99.80 | +| 13 | [RouterDC](https://arxiv.org/abs/2409.19886) [[GH]](https://github.com/shuhao02/RouterDC) | πŸŽ“ SUSTech | 33.75 | 39.84 | 73.00 | 49.05 | 10.75 | 97.60 | πŸŽ“ Academicβ€ƒβ€ƒπŸ’Ό Commercial  From 8e2b00385081f0deac226e19f1537cefd30faadd Mon Sep 17 00:00:00 2001 From: Jiarong Xing <42587491+jiarong0907@users.noreply.github.com> Date: Mon, 17 Nov 2025 21:46:53 -0600 Subject: [PATCH 02/10] Fix table formatting in README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 050cd6f2..8eb3406c 100644 --- a/README.md +++ b/README.md @@ -33,7 +33,7 @@ RouterArena bridges this gap by providing an open evaluation platform and benchm For more details, please see our [website](https://routeworks.github.io/leaderboard) and [blog](https://huggingface.co/blog/JerryPotter/who-routes-the-routers). | Rank | Router | Affiliation | Arena | Opt.Sel | Opt.Cost | Opt.Acc | Lat | Robust | -|------|---------|--------------|--------|------------|-----------|----------|----------|-------------| +|------|--------------------|-----------------------------|--------|------------|-----------|----------|----------|-------------| | πŸ₯‡ | [MIRT-BERT](https://arxiv.org/pdf/2506.01048) [[GH]](https://github.com/Mercidaiha/IRT-Router) | πŸŽ“ USTC | 66.89 | 3.44 | 19.62 | 78.18 | 27.03 | 94.50 | | πŸ₯ˆ | [Azure-Router](https://ai.azure.com/catalog/models/model-router) [[Web]](https://learn.microsoft.com/en-us/azure/ai-foundry/openai/concepts/model-router) | πŸ’Ό Microsoft | 66.66 | 22.52 | 46.32 | 81.96 | β€” | β€” | | πŸ₯‰ | [NIRT-BERT](https://arxiv.org/pdf/2506.01048) [[GH]](https://github.com/Mercidaiha/IRT-Router) | πŸŽ“ USTC | 66.12 | 3.83 | 14.04 | 77.88 | 10.42 | 44.50 | From 444f1aa50ff92c55f4a2011c3f670276fe00ff2c Mon Sep 17 00:00:00 2001 From: Jiarong Xing <42587491+jiarong0907@users.noreply.github.com> Date: Mon, 17 Nov 2025 21:58:22 -0600 Subject: [PATCH 03/10] test --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 8eb3406c..af2f8284 100644 --- a/README.md +++ b/README.md @@ -32,8 +32,8 @@ RouterArena bridges this gap by providing an open evaluation platform and benchm For more details, please see our [website](https://routeworks.github.io/leaderboard) and [blog](https://huggingface.co/blog/JerryPotter/who-routes-the-routers). -| Rank | Router | Affiliation | Arena | Opt.Sel | Opt.Cost | Opt.Acc | Lat | Robust | -|------|--------------------|-----------------------------|--------|------------|-----------|----------|----------|-------------| +| Rank | Router | Affiliation | Arena | Optimal Selection | Optimal Cost | Optimal Accuracy | Latency | Robustness | +|------|--------------------|-----------------------------|--------|-----------------|--------------|----------------|---------|------------| | πŸ₯‡ | [MIRT-BERT](https://arxiv.org/pdf/2506.01048) [[GH]](https://github.com/Mercidaiha/IRT-Router) | πŸŽ“ USTC | 66.89 | 3.44 | 19.62 | 78.18 | 27.03 | 94.50 | | πŸ₯ˆ | [Azure-Router](https://ai.azure.com/catalog/models/model-router) [[Web]](https://learn.microsoft.com/en-us/azure/ai-foundry/openai/concepts/model-router) | πŸ’Ό Microsoft | 66.66 | 22.52 | 46.32 | 81.96 | β€” | β€” | | πŸ₯‰ | [NIRT-BERT](https://arxiv.org/pdf/2506.01048) [[GH]](https://github.com/Mercidaiha/IRT-Router) | πŸŽ“ USTC | 66.12 | 3.83 | 14.04 | 77.88 | 10.42 | 44.50 | From 1777873104c3993c703b5b44949992ace0a6ed6a Mon Sep 17 00:00:00 2001 From: Jiarong Xing <42587491+jiarong0907@users.noreply.github.com> Date: Mon, 17 Nov 2025 22:07:58 -0600 Subject: [PATCH 04/10] test Updated leaderboard table with new rankings and links. --- README.md | 225 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 225 insertions(+) diff --git a/README.md b/README.md index af2f8284..45cd7728 100644 --- a/README.md +++ b/README.md @@ -50,6 +50,231 @@ For more details, please see our [website](https://routeworks.github.io/leaderbo πŸŽ“ Academicβ€ƒβ€ƒπŸ’Ό Commercial  + +
+ + + + + + + + + + + + + + + + + + + + + +
RankaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaRouterAffiliationArenaOpt.SelOpt.CostOpt.AccLatRobust
πŸ₯‡[RouterDC](https://arxiv.org/abs/2409.19886) [[GH]](https://github.com/shuhao02/RouterDC)πŸŽ“ USTC66.893.4419.6278.1827.0394.50
+ +
+ + +| Rank | Router | Affiliation | Arena | Optimal Selection | Optimal Cost | Optimal Accuracy | Latency | Robustness | +|------|--------------------|-----------------------------|--------|-----------------|--------------|----------------|---------|------------| +| πŸ₯‡ | [MIRT‑BERT](https://arxiv.org/pdf/2506.01048) [[GH]](https://github.com/Mercidaiha/IRT-Router) | πŸŽ“ USTC | 66.89 | 3.44 | 19.62 | 78.18 | 27.03 | 94.50 | +| πŸ₯ˆ | [Azure‑Router](https://ai.azure.com/catalog/models/model-router) [[Web]](https://learn.microsoft.com/en-us/azure/ai-foundry/openai/concepts/model-router) | πŸ’Ό Microsoft | 66.66 | 22.52 | 46.32 | 81.96 | β€” | β€” | +| πŸ₯‰ | [NIRT‑BERT](https://arxiv.org/pdf/2506.01048) [[GH]](https://github.com/Mercidaiha/IRT-Router) | πŸŽ“ USTC | 66.12 | 3.83 | 14.04 | 77.88 | 10.42 | 44.50 | +| 4 | [GPT‑5](https://openai.com/index/introducing-gpt-5/)| πŸ’Ό OpenAI | 64.32 | β€” | β€” | β€” | β€” | β€” | +| 5 | [vLLM‑SR](https://vllm-semantic-router.com/) [[GH]](https://github.com/vllm-project/semantic-router) [[HF]](https://huggingface.co/llm-semantic-router) | πŸ’Ό vLLM | 64.32 | 4.79 | 12.54 | 79.33 | 0.19 | 100.00 | +| 6 | [CARROT](https://arxiv.org/abs/2502.03261) [[GH]](https://github.com/somerstep/CARROT) [[HF]](https://huggingface.co/CARROT-LLM-Routing) | πŸŽ“ UMich | 63.87 | 2.68 | 6.77 | 78.63 | 1.50 | 93.60 | +| 7 | [Chayan](https://huggingface.co/adaptive-classifier/chayan) [[HF]](https://huggingface.co/adaptive-classifier/chayan) | πŸ’Ό Adaptive Classifier | 63.83 | 43.03 | 43.75 | 88.74 | β€” | β€” | +| 8 | [NotDiamond](https://www.notdiamond.ai/) | πŸ’Ό NotDiamond | 63.00 | 1.55 | 2.14 | 76.81 | β€” | β€” | +| 9 | [RouterBench‑MLP](https://arxiv.org/pdf/2403.12031) [[GH]](https://github.com/withmartian/routerbench) [[HF]](https://huggingface.co/datasets/withmartian/routerbench) | πŸŽ“ Academic | 57.56 | 13.39 | 24.45 | 83.32 | 90.91 | 96.90 | +| 10 | [GraphRouter](https://arxiv.org/abs/2410.03834) [[GH]](https://github.com/ulab-uiuc/GraphRouter) | πŸŽ“ UIUC | 57.22 | 4.73 | 38.33 | 74.25 | 2.70 | 97.50 | +| 11 | [RouterBench‑KNN](https://arxiv.org/pdf/2403.12031) [[GH]](https://github.com/withmartian/routerbench) [[HF]](https://huggingface.co/datasets/withmartian/routerbench) | πŸŽ“ Academic | 55.48 | 13.09 | 25.49 | 78.77 | 1.33 | 51.30 | +| 12 | [RouteLLM](https://arxiv.org/abs/2406.18665) [[GH]](https://github.com/lm-sys/RouteLLM) [[HF]](https://huggingface.co/routellm) | πŸŽ“ Berkeley | 48.07 | 99.72 | 99.63 | 68.76 | 0.40 | 99.80 | +| 13 | [RouterDC](https://arxiv.org/abs/2409.19886) [[GH]](https://github.com/shuhao02/RouterDC) | πŸŽ“ SUSTech | 33.75 | 39.84 | 73.00 | 49.05 | 10.75 | 97.60 | + + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
RankRouterAffiliationArenaOptimal SelectionOptimal CostOptimal AccuracyLatencyRobustness
πŸ₯‡MIRT-BERT [GH]πŸŽ“ USTC66.893.4419.6278.1827.0394.50
πŸ₯ˆAzure-Router [Web]πŸ’Ό Microsoft66.6622.5246.3281.96β€”β€”
πŸ₯‰NIRT-BERT [GH]πŸŽ“ USTC66.123.8314.0477.8810.4244.50
4GPT-5πŸ’Ό OpenAI64.32β€”β€”β€”β€”β€”
5vLLM-SR [GH] [HF]πŸ’Ό vLLM64.324.7912.5479.330.19100.00
6CARROT [GH] [HF]πŸŽ“ UMich63.872.686.7778.631.5093.60
7Chayan [HF]πŸ’Ό Adaptive Classifier63.8343.0343.7588.74β€”β€”
8NotDiamondπŸ’Ό NotDiamond63.001.552.1476.81β€”β€”
9RouterBench-MLP [GH] [HF]πŸŽ“ Academic57.5613.3924.4583.3290.9196.90
10GraphRouter [GH]πŸŽ“ UIUC57.224.7338.3374.252.7097.50
11RouterBench-KNN [GH] [HF]πŸŽ“ Academic55.4813.0925.4978.771.3351.30
12RouteLLM [GH] [HF]πŸŽ“ Berkeley48.0799.7299.6368.760.4099.80
13RouterDC [GH]πŸŽ“ SUSTech33.7539.8473.0049.0510.7597.60
+ +
+ + From c57092b4d8d492f9ed92fe2e47d740c4a54a5d9e Mon Sep 17 00:00:00 2001 From: Jiarong Xing <42587491+jiarong0907@users.noreply.github.com> Date: Mon, 17 Nov 2025 22:09:44 -0600 Subject: [PATCH 05/10] Revise leaderboard with updated router rankings Updated the leaderboard section with new rankings and links. --- README.md | 64 +------------------------------------------------------ 1 file changed, 1 insertion(+), 63 deletions(-) diff --git a/README.md b/README.md index 45cd7728..1ff89be6 100644 --- a/README.md +++ b/README.md @@ -32,69 +32,6 @@ RouterArena bridges this gap by providing an open evaluation platform and benchm For more details, please see our [website](https://routeworks.github.io/leaderboard) and [blog](https://huggingface.co/blog/JerryPotter/who-routes-the-routers). -| Rank | Router | Affiliation | Arena | Optimal Selection | Optimal Cost | Optimal Accuracy | Latency | Robustness | -|------|--------------------|-----------------------------|--------|-----------------|--------------|----------------|---------|------------| -| πŸ₯‡ | [MIRT-BERT](https://arxiv.org/pdf/2506.01048) [[GH]](https://github.com/Mercidaiha/IRT-Router) | πŸŽ“ USTC | 66.89 | 3.44 | 19.62 | 78.18 | 27.03 | 94.50 | -| πŸ₯ˆ | [Azure-Router](https://ai.azure.com/catalog/models/model-router) [[Web]](https://learn.microsoft.com/en-us/azure/ai-foundry/openai/concepts/model-router) | πŸ’Ό Microsoft | 66.66 | 22.52 | 46.32 | 81.96 | β€” | β€” | -| πŸ₯‰ | [NIRT-BERT](https://arxiv.org/pdf/2506.01048) [[GH]](https://github.com/Mercidaiha/IRT-Router) | πŸŽ“ USTC | 66.12 | 3.83 | 14.04 | 77.88 | 10.42 | 44.50 | -| 4 | [GPT-5](https://openai.com/index/introducing-gpt-5/)| πŸ’Ό OpenAI | 64.32 | β€” | β€” | β€” | β€” | β€” | -| 5 | [vLLM-SR](https://vllm-semantic-router.com/) [[GH]](https://github.com/vllm-project/semantic-router) [[HF]](https://huggingface.co/llm-semantic-router) | πŸ’Ό vLLM | 64.32 | 4.79 | 12.54 | 79.33 | 0.19 | 100.00 | -| 6 | [CARROT](https://arxiv.org/abs/2502.03261) [[GH]](https://github.com/somerstep/CARROT) [[HF]](https://huggingface.co/CARROT-LLM-Routing) | πŸŽ“ UMich | 63.87 | 2.68 | 6.77 | 78.63 | 1.50 | 93.60 | -| 7 | [Chayan](https://huggingface.co/adaptive-classifier/chayan) [[HF]](https://huggingface.co/adaptive-classifier/chayan) | πŸ’Ό Adaptive Classifier | 63.83 | 43.03 | 43.75 | 88.74 | β€” | β€” | -| 8 | [NotDiamond](https://www.notdiamond.ai/) | πŸ’Ό NotDiamond | 63.00 | 1.55 | 2.14 | 76.81 | β€” | β€” | -| 9 | [RouterBench-MLP](https://arxiv.org/pdf/2403.12031) [[GH]](https://github.com/withmartian/routerbench) [[HF]](https://huggingface.co/datasets/withmartian/routerbench) | πŸŽ“ Academic | 57.56 | 13.39 | 24.45 | 83.32 | 90.91 | 96.90 | -| 10 | [GraphRouter](https://arxiv.org/abs/2410.03834) [[GH]](https://github.com/ulab-uiuc/GraphRouter) | πŸŽ“ UIUC | 57.22 | 4.73 | 38.33 | 74.25 | 2.70 | 97.50 | -| 11 | [RouterBench-KNN](https://arxiv.org/pdf/2403.12031) [[GH]](https://github.com/withmartian/routerbench) [[HF]](https://huggingface.co/datasets/withmartian/routerbench) | πŸŽ“ Academic | 55.48 | 13.09 | 25.49 | 78.77 | 1.33 | 51.30 | -| 12 | [RouteLLM](https://arxiv.org/abs/2406.18665) [[GH]](https://github.com/lm-sys/RouteLLM) [[HF]](https://huggingface.co/routellm) | πŸŽ“ Berkeley | 48.07 | 99.72 | 99.63 | 68.76 | 0.40 | 99.80 | -| 13 | [RouterDC](https://arxiv.org/abs/2409.19886) [[GH]](https://github.com/shuhao02/RouterDC) | πŸŽ“ SUSTech | 33.75 | 39.84 | 73.00 | 49.05 | 10.75 | 97.60 | - -πŸŽ“ Academicβ€ƒβ€ƒπŸ’Ό Commercial  - - -
- - - - - - - - - - - - - - - - - - - - - -
RankaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaRouterAffiliationArenaOpt.SelOpt.CostOpt.AccLatRobust
πŸ₯‡[RouterDC](https://arxiv.org/abs/2409.19886) [[GH]](https://github.com/shuhao02/RouterDC)πŸŽ“ USTC66.893.4419.6278.1827.0394.50
- -
- - -| Rank | Router | Affiliation | Arena | Optimal Selection | Optimal Cost | Optimal Accuracy | Latency | Robustness | -|------|--------------------|-----------------------------|--------|-----------------|--------------|----------------|---------|------------| -| πŸ₯‡ | [MIRT‑BERT](https://arxiv.org/pdf/2506.01048) [[GH]](https://github.com/Mercidaiha/IRT-Router) | πŸŽ“ USTC | 66.89 | 3.44 | 19.62 | 78.18 | 27.03 | 94.50 | -| πŸ₯ˆ | [Azure‑Router](https://ai.azure.com/catalog/models/model-router) [[Web]](https://learn.microsoft.com/en-us/azure/ai-foundry/openai/concepts/model-router) | πŸ’Ό Microsoft | 66.66 | 22.52 | 46.32 | 81.96 | β€” | β€” | -| πŸ₯‰ | [NIRT‑BERT](https://arxiv.org/pdf/2506.01048) [[GH]](https://github.com/Mercidaiha/IRT-Router) | πŸŽ“ USTC | 66.12 | 3.83 | 14.04 | 77.88 | 10.42 | 44.50 | -| 4 | [GPT‑5](https://openai.com/index/introducing-gpt-5/)| πŸ’Ό OpenAI | 64.32 | β€” | β€” | β€” | β€” | β€” | -| 5 | [vLLM‑SR](https://vllm-semantic-router.com/) [[GH]](https://github.com/vllm-project/semantic-router) [[HF]](https://huggingface.co/llm-semantic-router) | πŸ’Ό vLLM | 64.32 | 4.79 | 12.54 | 79.33 | 0.19 | 100.00 | -| 6 | [CARROT](https://arxiv.org/abs/2502.03261) [[GH]](https://github.com/somerstep/CARROT) [[HF]](https://huggingface.co/CARROT-LLM-Routing) | πŸŽ“ UMich | 63.87 | 2.68 | 6.77 | 78.63 | 1.50 | 93.60 | -| 7 | [Chayan](https://huggingface.co/adaptive-classifier/chayan) [[HF]](https://huggingface.co/adaptive-classifier/chayan) | πŸ’Ό Adaptive Classifier | 63.83 | 43.03 | 43.75 | 88.74 | β€” | β€” | -| 8 | [NotDiamond](https://www.notdiamond.ai/) | πŸ’Ό NotDiamond | 63.00 | 1.55 | 2.14 | 76.81 | β€” | β€” | -| 9 | [RouterBench‑MLP](https://arxiv.org/pdf/2403.12031) [[GH]](https://github.com/withmartian/routerbench) [[HF]](https://huggingface.co/datasets/withmartian/routerbench) | πŸŽ“ Academic | 57.56 | 13.39 | 24.45 | 83.32 | 90.91 | 96.90 | -| 10 | [GraphRouter](https://arxiv.org/abs/2410.03834) [[GH]](https://github.com/ulab-uiuc/GraphRouter) | πŸŽ“ UIUC | 57.22 | 4.73 | 38.33 | 74.25 | 2.70 | 97.50 | -| 11 | [RouterBench‑KNN](https://arxiv.org/pdf/2403.12031) [[GH]](https://github.com/withmartian/routerbench) [[HF]](https://huggingface.co/datasets/withmartian/routerbench) | πŸŽ“ Academic | 55.48 | 13.09 | 25.49 | 78.77 | 1.33 | 51.30 | -| 12 | [RouteLLM](https://arxiv.org/abs/2406.18665) [[GH]](https://github.com/lm-sys/RouteLLM) [[HF]](https://huggingface.co/routellm) | πŸŽ“ Berkeley | 48.07 | 99.72 | 99.63 | 68.76 | 0.40 | 99.80 | -| 13 | [RouterDC](https://arxiv.org/abs/2409.19886) [[GH]](https://github.com/shuhao02/RouterDC) | πŸŽ“ SUSTech | 33.75 | 39.84 | 73.00 | 49.05 | 10.75 | 97.60 | - -
@@ -274,6 +211,7 @@ For more details, please see our [website](https://routeworks.github.io/leaderbo +πŸŽ“ Academicβ€ƒβ€ƒπŸ’Ό Commercial  + +
+ +
+ + +| Rank | Router | Affiliation | Arena | Optimal Selection | Optimal Cost | Optimal Accuracy | Latency | Robustness | +|------|--------------------|-----------------------------|--------|-----------------|--------------|----------------|---------|------------| +| πŸ₯‡ | [MIRT‑BERT](https://arxiv.org/pdf/2506.01048) [[GH]](https://github.com/Mercidaiha/IRT-Router) | πŸŽ“ USTC | 66.89 | 3.44 | 19.62 | 78.18 | 27.03 | 94.50 | +| πŸ₯ˆ | [Azure‑Router](https://ai.azure.com/catalog/models/model-router) [[Web]](https://learn.microsoft.com/en-us/azure/ai-foundry/openai/concepts/model-router) | πŸ’Ό Microsoft | 66.66 | 22.52 | 46.32 | 81.96 | β€” | β€” | +| πŸ₯‰ | [NIRT‑BERT](https://arxiv.org/pdf/2506.01048) [[GH]](https://github.com/Mercidaiha/IRT-Router) | πŸŽ“ USTC | 66.12 | 3.83 | 14.04 | 77.88 | 10.42 | 44.50 | +| 4 | [GPT‑5](https://openai.com/index/introducing-gpt-5/)| πŸ’Ό OpenAI | 64.32 | β€” | β€” | β€” | β€” | β€” | +| 5 | [vLLM‑SR](https://vllm-semantic-router.com/) [[GH]](https://github.com/vllm-project/semantic-router) [[HF]](https://huggingface.co/llm-semantic-router) | πŸ’Ό vLLM | 64.32 | 4.79 | 12.54 | 79.33 | 0.19 | 100.00 | +| 6 | [CARROT](https://arxiv.org/abs/2502.03261) [[GH]](https://github.com/somerstep/CARROT) [[HF]](https://huggingface.co/CARROT-LLM-Routing) | πŸŽ“ UMich | 63.87 | 2.68 | 6.77 | 78.63 | 1.50 | 93.60 | +| 7 | [Chayan](https://huggingface.co/adaptive-classifier/chayan) [[HF]](https://huggingface.co/adaptive-classifier/chayan) | πŸ’Ό Adaptive Classifier | 63.83 | 43.03 | 43.75 | 88.74 | β€” | β€” | +| 8 | [NotDiamond](https://www.notdiamond.ai/) | πŸ’Ό NotDiamond | 63.00 | 1.55 | 2.14 | 76.81 | β€” | β€” | +| 9 | [RouterBench‑MLP](https://arxiv.org/pdf/2403.12031) [[GH]](https://github.com/withmartian/routerbench) [[HF]](https://huggingface.co/datasets/withmartian/routerbench) | πŸŽ“ Academic | 57.56 | 13.39 | 24.45 | 83.32 | 90.91 | 96.90 | +| 10 | [GraphRouter](https://arxiv.org/abs/2410.03834) [[GH]](https://github.com/ulab-uiuc/GraphRouter) | πŸŽ“ UIUC | 57.22 | 4.73 | 38.33 | 74.25 | 2.70 | 97.50 | +| 11 | [RouterBench‑KNN](https://arxiv.org/pdf/2403.12031) [[GH]](https://github.com/withmartian/routerbench) [[HF]](https://huggingface.co/datasets/withmartian/routerbench) | πŸŽ“ Academic | 55.48 | 13.09 | 25.49 | 78.77 | 1.33 | 51.30 | +| 12 | [RouteLLM](https://arxiv.org/abs/2406.18665) [[GH]](https://github.com/lm-sys/RouteLLM) [[HF]](https://huggingface.co/routellm) | πŸŽ“ Berkeley | 48.07 | 99.72 | 99.63 | 68.76 | 0.40 | 99.80 | +| 13 | [RouterDC](https://arxiv.org/abs/2409.19886) [[GH]](https://github.com/shuhao02/RouterDC) | πŸŽ“ SUSTech | 33.75 | 39.84 | 73.00 | 49.05 | 10.75 | 97.60 | + +
@@ -211,7 +274,6 @@ For more details, please see our [website](https://routeworks.github.io/leaderbo -πŸŽ“ Academicβ€ƒβ€ƒπŸ’Ό Commercial  - -
- -
- - | Rank | Router | Affiliation | Arena | Optimal Selection | Optimal Cost | Optimal Accuracy | Latency | Robustness | |------|--------------------|-----------------------------|--------|-----------------|--------------|----------------|---------|------------| | πŸ₯‡ | [MIRT‑BERT](https://arxiv.org/pdf/2506.01048) [[GH]](https://github.com/Mercidaiha/IRT-Router) | πŸŽ“ USTC | 66.89 | 3.44 | 19.62 | 78.18 | 27.03 | 94.50 | @@ -95,186 +49,6 @@ For more details, please see our [website](https://routeworks.github.io/leaderbo | 13 | [RouterDC](https://arxiv.org/abs/2409.19886) [[GH]](https://github.com/shuhao02/RouterDC) | πŸŽ“ SUSTech | 33.75 | 39.84 | 73.00 | 49.05 | 10.75 | 97.60 | -
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
RankRouterAffiliationArenaOptimal SelectionOptimal CostOptimal AccuracyLatencyRobustness
πŸ₯‡MIRT-BERT [GH]πŸŽ“ USTC66.893.4419.6278.1827.0394.50
πŸ₯ˆAzure-Router [Web]πŸ’Ό Microsoft66.6622.5246.3281.96β€”β€”
πŸ₯‰NIRT-BERT [GH]πŸŽ“ USTC66.123.8314.0477.8810.4244.50
4GPT-5πŸ’Ό OpenAI64.32β€”β€”β€”β€”β€”
5vLLM-SR [GH] [HF]πŸ’Ό vLLM64.324.7912.5479.330.19100.00
6CARROT [GH] [HF]πŸŽ“ UMich63.872.686.7778.631.5093.60
7Chayan [HF]πŸ’Ό Adaptive Classifier63.8343.0343.7588.74β€”β€”
8NotDiamondπŸ’Ό NotDiamond63.001.552.1476.81β€”β€”
9RouterBench-MLP [GH] [HF]πŸŽ“ Academic57.5613.3924.4583.3290.9196.90
10GraphRouter [GH]πŸŽ“ UIUC57.224.7338.3374.252.7097.50
11RouterBench-KNN [GH] [HF]πŸŽ“ Academic55.4813.0925.4978.771.3351.30
12RouteLLM [GH] [HF]πŸŽ“ Berkeley48.0799.7299.6368.760.4099.80
13RouterDC [GH]πŸŽ“ SUSTech33.7539.8473.0049.0510.7597.60
- -
- - From 411a2a8a18e078d651ceb654799dafb9b704f56c Mon Sep 17 00:00:00 2001 From: Jiarong Xing <42587491+jiarong0907@users.noreply.github.com> Date: Mon, 17 Nov 2025 22:16:05 -0600 Subject: [PATCH 08/10] Add academic and commercial labels to README --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 8576b4e9..5231f32c 100644 --- a/README.md +++ b/README.md @@ -48,6 +48,7 @@ For more details, please see our [website](https://routeworks.github.io/leaderbo | 12 | [RouteLLM](https://arxiv.org/abs/2406.18665) [[GH]](https://github.com/lm-sys/RouteLLM) [[HF]](https://huggingface.co/routellm) | πŸŽ“ Berkeley | 48.07 | 99.72 | 99.63 | 68.76 | 0.40 | 99.80 | | 13 | [RouterDC](https://arxiv.org/abs/2409.19886) [[GH]](https://github.com/shuhao02/RouterDC) | πŸŽ“ SUSTech | 33.75 | 39.84 | 73.00 | 49.05 | 10.75 | 97.60 | +πŸŽ“ Academicβ€ƒβ€ƒπŸ’Ό Commercial