[AIGNLAI]

Summary

• Add xRouteBench, MMR-Bench V2 and R2-Bench through the existing BaseDatasetLoader contract; router interfaces, training entry point and shared metrics are unchanged.
• Provide benchmark configs, frozen source/model/budget registries, aligned model and TRouter task descriptions, README News and a detailed protocol guide.
• Validate identities, scores, costs, token counts and asset paths; preserve raw costs, group repeated inputs across splits, and audit missing/failed outcomes without zero imputation. Record source revisions, candidate identities, coverage, train/test hashes and training-only cost scaling in protocol sidecars.
• Adapt MMR multi-image inputs to ordered, content-addressed contact sheets. Represent R2's released model-budget actions as ordinary candidates, preserving physical-model and nominal-budget metadata.
• Fix RouteLLM SW/MF/BERT automatic strong/weak selection for tied training means and partially configured endpoints. Preserve explicit endpoints and reject invalid/same-model pairs.
• Add an all-registered-method runtime matrix to existing CI and reusable smoke scripts with retained logs, result JSONs and substitution metadata.

Validation

• python -m unittest discover -s tests -q: 67 tests passed, including 30 methods × 3 offline benchmark fixtures plus 24 compatible MMR multimodal cases. Router training, prediction, allocation, nAUC/RCI and result writing are actually executed.
• Bounded real-data smoke after the RouteLLM fix: 90/90 passed, covering all 30 registered methods on all three new benchmarks, without explicitly overriding RouteLLM endpoints.
• Offline smoke using automatic RouteLLM endpoints and the existing local RouteFM BGE router checkpoint: 90/90 passed.
• MMR multimodal fixture smoke: 24/24 passed.
• Python compilation and git diff --check passed.

Scope of smoke validation

These are CPU runtime/integration checks, not full-data scientific reproductions. They use input-only hash features and one training epoch. BERT and ModelSAT execute real tiny locally initialized Transformer models (including ModelSAT LoRA), not their production pretrained weights. NIRT's paid auxiliary-LLM annotation boundary is replaced with an offline response; UMAP/HDBSCAN, training and prediction still execute. Real-data RouteFM smoke loads the existing local released BGE router weights with dimension-matched hash features; portable CI instead uses an untrained tiny official-architecture checkpoint. MMR real-record smoke is text-only; multimodal plumbing uses offline image fixtures.

Benchmark protocol limitations

• MMR-Bench V2 needs separately prepared upstream image assets. Its released costs are output-side reference estimates; common-success-support filtering and its coverage are explicit.
• R2 uses actual recorded output tokens and frozen reference prices, not nominal budgets or full input-inclusive bills. Best Single/RCI are action-level (model × budget), not physical-model-level. The full release is approximately 25 GB and was not downloaded for smoke.
• xRoute's visual/video scenarios are explicitly text-view routing on released query descriptions. Pairwise personalized-preference artifacts are rejected rather than coerced into performance/cost labels.
• Invalid explicit RouteLLM pairs and tie_policy=drop pools without non-tied preference labels still fail clearly; no labels are fabricated to make training pass.

See docs/new_benchmarks.md for source pins, configuration, offline layouts, cost/split protocols and smoke commands.