Skip to content

feat(scheduler): Cost-Aware & Latency-Aware Dynamic Multi-Provider Routing #19

Description

@sdageltc

Summary & Motivation

Different coding tasks require different levels of model capability. Running high-end frontier models (Claude 3.5 Sonnet / GPT-4o) on simple boilerplate tasks wastes budget, while running lightweight models on complex architectural tasks leads to repeated failed attempts.

We want to expand orchestrator/provider_scheduler.py into a Cost- & Risk-Tiered Dynamic Provider Router.


Proposed Routing Logic

  1. Risk-Tiered Model Selection:
    • Tier 1 (Trivial / Boilerplate / Format): Gemini 2.5 Flash Lite / Qwen 2.5 Coder (Fast, sub-cent cost).
    • Tier 2 (Standard Task Implementation / Unit Tests): Gemini 2.5 Flash / Claude 3.5 Haiku / GPT-4o-mini.
    • Tier 3 (Crucible / Architecture / High-Risk Refactor / 3-Strike Repair): Claude 3.5 Sonnet / GPT-4o / DeepSeek R1.
  2. Dynamic Fallback Escalation:
    • If a contract fails Attempt 1 with a Tier 2 model, Attempt 2 automatically escalates to a Tier 3 frontier model with the failed attempt evidence.
  3. Cost Ceiling Enforcement:
    • Respects max cost limits configured in BudgetGuard.

Acceptance Criteria

  • Routes contract prompts based on declared risk_tier or failure history.
  • Seamless fallback on provider rate limits (HTTP 429) or timeouts.
  • Full simulation unit tests in tests/test_provider_scheduler.py.

Metadata

Metadata

Assignees

No one assigned

    Labels

    architectureCore control loop and system architectureenhancementNew feature or requestintegrationsWorker adapters, LLM providers, and external tools

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions