Model router with preference learning data capturing

[laminair]

What the router does, conceptually

The router picks the coding model (Claude → Anthropic tiers; OpenCode → whatever providers you have configured, including a local Ollama qwen3:27b) per session and per turn, and learns from the user's own behavior which model serves them best for which kind of task. The summarizer stays one fixed cheap model — it gets a second job as the judge that produces a structured verdict on every committed turn, on the same worker thread that already writes the commit summary (no extra round-trip when the trace contains an obvious "no, that's wrong").

The shape mirrors how an experienced developer would choose a model: try a model, see what happened, prefer the one that paid off in situations like this. The user never has to rate anything; corrections, re-prompts, discards and reverts are all signals. Explicit 1-5 ratings (Ctrl-G rate or the dashboard) are the user's most direct vote.

Where the signal comes from

Every aGiTrack turn already produces the interaction trace the commit carries (the same ## User / ## Agent text). On the existing summary worker, the judge reads that trace and returns a tight JSON verdict:

{ "task_class": "debug", "complexity": "medium",
  "correction": "explicit_negative", "evidence": "wrong file" }

A loud negative trace ("no, that's wrong — undo that") is caught by a substring heuristic before the model call so the small model isn't even invoked for the obvious cases. The harder cases go to the model. The verdict feeds three things:

• the policy score for the current (model, task_class) cell (EMA blending at ≥3 per-class samples so per-class EMAs dominate the model-wide one once you have enough data);
• the router's hint when a different model would score better (visible in the status bar, Ctrl-G → model lets the user switch);
• the audit feed on the new /routing dashboard panel, alongside the per-model quality table and a rating widget.

Where the model signal comes from

A turn records exactly one (or more) of these:

Signal	Source	Weight
explicit rating (1-5)	Ctrl-G → rate or /routing	strongest
judge: explicit negative / redo	the summarizer-as-judge	strong negative
judge: clean turn	the summarizer-as-judge	weak positive
discard (Esc → "Yes, discard")	the runner	strongest negative — the user threw the work away
revert (git revert of an <aGiTrack> commit)	new collect.py pass	strong negative
redo_followup (re-prompt)	the judge reads the queued follow-up	medium
post_agent_edit (user patches the worktree right after a turn)	the runner	weak — could be unrelated
cancel (Esc, kept the work)	the runner	medium negative
switch / reroute	the runner	neutral — the next turn's rating is the real signal

Each is folded into the per-model and per-(model, task_class) quality via an exponential moving average (alpha 0.3, quality in [0, 1], cold-start prior 0.55). The event ring buffer caps at 500 so the file stays bounded.

How a decision is made

ModelRouter.choose() scores every model in the user's pool (auto-seeded from the active backend's own model list: Claude → the curated tiers; OpenCode → whatever the user has configured there, including a local Ollama provider):

final = sample(quality_ema, exploration/total_attempts) − λ · cost_tier

with a Thompson-sampling bonus that shrinks as attempts grow (so unknown models are still tried occasionally, well-trodden models are picked deterministically). With ≥3 samples for a (model, task_class) cell, the per-class EMA dominates the model-wide one (70/30 blend) — the router learns "haiku is great at small edits, opus at big refactors" from a few sessions of normal use.

Three modes:

• off (default) — the judge still runs and records signals, but the router never switches models. Lets a curious user build a profile without surprises.
• suggest — status-bar hint when another model would score better; Ctrl-G → model opens a pool picker with a "router suggests" badge; switching happens on user demand.
• auto — between turns, when the agent is idle, the runner types the backend's own model-switch command into the TUI (/model <name> for Claude, ctrl+x m then typing for OpenCode's model dialog), with a relaunch-with---model fallback when the picker UI can't be scripted deterministically. Per-turn auto-routing in headless json/bridge mode too.

Why the summarizer-as-judge

The summarizer is already running on every committed turn in every mode (proxy async, background synchronous, shell inline). It's a cheap model, it's in bare mode with no tools or memory, and it shares the summary worker's thread and lock — so adding a second bare call costs "one small completion per turn" and nothing else. The judge shares the same backend, the same model, the same scratch dir, and the same UnusableSummaryError / prompt-echo guards as the commit summary, so a tiny local model that echoes its prompt is detected and recorded as usable=False (no signal) rather than corrupting the store.

Pool topology

Per the user's design choice: the pool is derived from the active backend, never a global "which pool" question. The router never asks "which pool" — when you're on Claude, the pool is Anthropic tiers; when you're on OpenCode, the pool is whatever providers you've configured there, including a local Ollama qwen3:27b. Cross-backend pool mixing was deliberately not done; it would require either new ProxyAgent machinery per local HTTP backend or a heavy OpenCode-everything push.

What ships in v0.6.0.dev1

• New package agitrack/routing/ (~1,500 lines): store, judge, policy, router facade, signals, switch — all best-effort by contract.
• New dashboard panel at /routing: per-model quality table, event feed, rating widget, sync toggle.
• Two new Ctrl-G commands: model (pool picker, router-suggested badge) and rate (1-5).
• New settings-menu entries: routing_mode, routing_pool (JSON), routing_allow_cloud, routing_judge, routing_sync.
• Judge wired into the proxy, background, and shell summary workers; in-TUI model switching from the runner.
• Implicit signals wired at the existing discard / cancel sites.
• New per-user store .agitrack/routing.json (git-ignored, per-GitHub-user keyed) with optional orphan-ref sync to refs/agitrack/routing-prefs (mirrors the learn-page pattern).
• 67 new tests (store / judge / policy / switch / router facade / signal recorders / runner integration / dashboard endpoints / real-CLI smoke) — bash scripts/check.sh green, 2201 tests pass.

Why a persistent signal store, not a volatile cache

The store follows the same shape as the learn page's learning.json: per-user, git-ignored, optional orphan-ref sync so preferences follow the user across machines. A fresh clone restores your profile the first time you open /routing. Empty local profile never overwrites a synced remote; the local store is the source of truth, rebuilt on every push.

Risks called out

• In-TUI model switch is version-sensitive: the documented /model <name> and ctrl+x m sequences are verified against current Claude and OpenCode releases, but a future backend TUI revision could break the byte sequence. The runner falls back to relaunch-with---model when injection can't be verified, so a drift degrades to "auto mode" being a relaunch instead of a mid-session swap — never a crash, never a stuck model.
• Judge on a tiny local model is noisy: the guard chain (heuristic → model → strict JSON schema → echo detection) reduces the noise, but the router is intentionally conservative in auto mode (a default min_margin of 0.05 keeps marginal wins from triggering disruptive mid-session switches).
• The judge adds a small token cost per turn: a 12k-char trace cap, a 1.5k-char response cap, and a 90s bare-run timeout. The summarizer's existing cost accounting is extended so the judge line items land on the commit's summary_tokens_* fields alongside the commit summary's. Users who don't want the router to learn anything can set routing_judge: false to silence the call while keeping the router scaffolding.

Demo flow

1. agitrack — open a session in claude (or opencode), coding on dev.
2. Use the agent for a turn, give it a 5-star rating (Ctrl-G → rate, or on /routing). The signal lands in .agitrack/routing.json under your GitHub id.
3. Ctrl-G → model shows the pool with a "router suggests" badge on the model that scores highest for the kind of task the last turn was.
4. Switch to a local Ollama qwen3:27b. aGiTrack types /model ollama/qwen3:27b into the TUI; the backend updates the model line, the next turn runs on qwen.
5. The qwen turn lands a commit; the runner records the verdict; the next Ctrl-G → model call already sees a richer profile (debug: claude wins, small edits: qwen is fine).
6. On a different machine, the synced profile is restored from refs/agitrack/routing-prefs the first time you open /routing.

[laminair]

@shiqiangw The above is a very early draft of what a router could look like. We might be able to generate preference labels, scores/rewards along the way without much overhead.

[shiqiangw]

@laminair Could you change the version back to the original? Version bump should happen automatically via git workflow. The current version tag causes Windows CI to fail.

[shiqiangw]

Also, does the routing consider token caches, which will need a new write when switching to a new model? And how is the cost tier in the expression determined?

[laminair]

@shiqiangw Yes, that was unintentional.

The PR draft is a very, very early version. There's no token, caching, or memory accounting built-in yet. The goal for me is to first figure out how we can collect data that we can use for the routing decision. Once that is solved we should be able to factor in token economics.
My assumption for the moment is that we, by default, rely on a small (local) model and only escalate to cloud models when needed.