Skip to content

Remove RCI metric from routing evaluation - #17

Merged
AIGNLAI merged 1 commit into
mainfrom
fix/remove-rci
Oct 8, 2026
Merged

AIGNLAI merged 1 commit into
mainfrom
fix/remove-rci

Conversation

@AIGNLAI

@AIGNLAI AIGNLAI commented Oct 8, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Remove BaseRouter.cal_rci and all RCI calls, logs, and diagnostic-only allocation tracking from shared, Carrot, EA-RAM, and reward-model evaluation paths.
  • Remove RCI documentation and benchmark protocol references; retain nAUC, Peak Score, QNC, allocation rules, and the existing Pareto JSON schema.
  • Update smoke checks to validate finite Peak Score instead of RCI; preserve EA-RAM null-action coverage with a real-auction regression test.

Validation

  • Rebased onto latest main after PR Add xRouteBench, MMR-Bench V2 and R2-Bench; fix RouteLLM pair selection #16 merged; this PR contains only RCI-removal changes.
  • Full unittest discovery: 67 tests passed, including the 30-method x 3-benchmark offline matrix and 24 compatible multimodal method checks.
  • Persistent offline smoke: 90/90 passed.
  • Bounded real-data smoke: 90/90 passed across XRouteBench, MMRBenchV2, and R2Bench.
  • All 180 persistent smoke results have identical Pareto curves and nAUC to the corresponding pre-removal results; new logs contain no RCI output.
  • Syntax compilation and git diff --check passed.

Smoke runs exercise real router training, prediction, allocation, evaluation, and JSON output. They use small samples, deterministic hash embeddings, tiny local transformer checkpoints, and a stub at the paid NIRT annotation boundary; RouteFM uses existing released local weights. These are runtime/regression checks, not full benchmark quality claims.

Compatibility

The deprecated cal_rci API is intentionally removed. Existing routing curves and result JSON formats are unchanged.

@AIGNLAI
AIGNLAI merged commit fb875be into main Oct 8, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant