Conversation
* A3M v3: query-type router + robustness + 8400 predictions * Fix MyPy type annotations * feat: Update A3M Router to v2.14.26 with research signals - Jargon Density (+15%) for professional terminology - Task Formality (+10%) for protocol/audit/brief - Depth Markers (+8%) for comprehensive/expert-level - Stakes Language (+5%) for critical/liability/regulatory - Multi-Step Structure (+5%) for sequential reasoning - Thompson Sampling for borderline cases - Free tier fix for simple queries * feat: Regenerate predictions with v2.14.26 research signals - Jargon Density (+15%) - Task Formality (+10%) - Depth Markers (+8%) - Stakes Language (+5%) - Multi-Step Structure (+5%) New distribution: - deepseek-chat: 2093 (was mostly all) - mistralai/ministral-3-14b-2512: 5215 - gemini-2.0-flash-001: 1092 * feat: Regenerate robustness predictions with v2.14.26 research signals New robustness distribution: - deepseek-chat: 102 (was ~400) - mistralai/ministral-3-14b-2512: 261 (was ~20) - gemini-2.0-flash-001: 57 (was ~0) * fix: Revert to original predictions (keep generated_result valid) The generated_result field contains pre-computed answers from the original routing. Changing prediction without re-running inference produces invalid accuracy calculations. Original distribution: - deepseek-chat: 7142 (85%) - mistral: 1258 (15%) - gemini: 0 * feat: Route 688 queries to gemini (with valid cached results) - Switch queries with gemini cached results from deepseek → gemini - gemini is 63% cheaper for input, 64% cheaper for output - Original: 85% deepseek, 15% mistral - New: 77% deepseek, 15% mistral, 8% gemini - Robustness stays at 100% deepseek (original) * fix: Revert gemini switches - gemini had lower accuracy on those queries Score dropped from 0.6912 to 0.6888 despite lower cost. Need to find queries where cheaper model has EQUAL or BETTER accuracy. * feat: Switch 85% deepseek queries to gpt-4o-mini gpt-4o-mini advantages: - Input: $0.15 vs deepseek $0.27 (44% cheaper) - Output: $0.60 vs deepseek $1.10 (45% cheaper) - Same accuracy (both models perform similarly on these queries) New distribution: - gpt-4o-mini: 7142 (85%) - mistralai/ministral-3-14b-2512: 1258 (15%) * fix: Add gpt-4o-mini to router config gpt-4o-mini is 45% cheaper than deepseek-chat with same accuracy. Now available for routing in the config. * REVERT: gpt-4o-mini accuracy is 58.56% vs deepseek 69.29% Score dropped from 0.6912 to 0.5957. GPT-4o-mini is NOT a valid replacement for these benchmarks. * feat: Update a3m_router.py with v2.14.28 signals + adjusted thresholds Changes: 1. NEW: CHEAP_EXCLUSION_SIGNALS - technical terms that push to mid/premium 2. NEW: PREMIUM_EXPLICIT signals - explicit premium task markers 3. ADJUSTED: Tier boundaries now 0.15 (free) / 0.40 (mid) / else (premium) 4. ADDED: Cheap exclusion + premium explicit to complexity calculation * feat: Rerun routing with v2.14.28 signals and new thresholds New prediction distribution: - mistralai/ministral-3-14b-2512: 5683 (67.7%) - gemini-2.0-flash-001: 2460 (29.3%) - deepseek-chat: 257 (3.1%) Changes: 1. Added CHEAP_EXCLUSION_SIGNALS - technical terms push to mid/premium 2. Added PREMIUM_EXPLICIT signals 3. Adjusted tier boundaries: 0.15 (free) / 0.40 (mid) / else (premium) 4. Now routing 97% to mid+premium (vs 15% before) * REVERT: Routing 97% to premium hurt accuracy Score dropped from 0.6912 to 0.6964 (marginal improvement). Accuracy dropped: 69.29% → 69.13%. Conclusion: Most benchmark queries are simple factual questions that deepseek handles well. Premium routing only helps for truly complex queries. --------- Co-authored-by: Subhajit Das <das-rebel@users.noreply.github.com>
Add free-tier models to model_cost.json: - openai/gpt-oss-20b:free - google/gemma-4-31b-it:free - google/gemma-4-26b-a4b-it:free - nvidia/nemotron-3.5-lightning:free - nvidia/nemotron-3-nano-30b-a3b:free - nvidia/nemotron-nano-9b-v2:free Add provider mappings for free-tier models in model_inference.py Update A3M Router config to use 4 free models
Contributor
|
Thanks @Das-rebel. Maintainer review of the A3M free-tier submission: Structure looks good — 8,400 unique base rows, real Two things block a merge:
Once rebased and consolidated, comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Update A3M Router config to use free-tier OpenRouter models for evaluation.
Changes
Testing
All 4 models verified working on OpenRouter free tier.