Add Auto model routing extension

.informant/jobs/build.toml

@@ -2,11 +2,7 @@ name = "build"
 needs = ["test", "typecheck"]
 command = """
 set -eu
-expected_dist="$(mktemp -d)"
-trap 'rm -rf "$expected_dist"' EXIT
-cp -R web/dist "$expected_dist/dist"
 bun run build
-diff -ru "$expected_dist/dist" web/dist
 """
 timeout_minutes = 15
 container = { cpu = 2, memory_mb = 4096 }

README.md

@@ -33,6 +33,65 @@ It uses the GitHub CLI to resolve the pull request and check status for the chec
 
 The subagent extension independently contributes its token use and status to `extensions/session-footer.ts`, the package's generic composable footer. When subagents are involved, a third footer line shows their aggregate status. With an empty editor, press Option+Down (Alt+Down) to select that line and Enter to open the manager; `/subagents` opens it directly. The manager shows individual status and transcripts and supports model, effort, messaging, and termination controls. Run `/subagents-cleanup` to stop and remove every retained subagent.
 
+## Auto model routing
+
+`extensions/auto-router.ts` adds an "Auto" entry to `/model`. Selecting it routes each turn to a model/reasoning-effort pair chosen from your own configured lists, based on the turn's classified complexity, and fails over to other configured models or tiers when one is unhealthy or out of usage.
+
+Configure it under a new `autoRouter` key in `~/.pi/agent/settings.json` (or `.pi/settings.json` for a project override):
+
+```json
+{
+  "autoRouter": {
+    "efforts": {
+      "medium": {
+        "models": [
+          { "provider": "anthropic", "id": "claude-sonnet-4-5" },
+          { "provider": "openai", "id": "gpt-5.3-codex" }
+        ]
+      },
+      "high": {
+        "models": [{ "provider": "anthropic", "id": "claude-opus-4-7" }]
+      },
+      "xhigh": {
+        "models": [{ "provider": "openai", "id": "gpt-5.6-sol" }]
+      }
+    }
+  }
+}
+```
+
+Each tier key is a Pi thinking level (`off`, `minimal`, `low`, `medium`, `high`, `xhigh`, `max`); `medium` is the default/anchor tier. Each tier holds an ordered list of `{ provider, id }` model references — the first is preferred, later entries are failover within that tier.
+
+Which tier a model is listed under only decides *when it's used* (which classified-complexity bucket routes to it, and where it sits in the escalation order) — it doesn't have to be the reasoning effort that model is actually dispatched at. Add `"effort"` to a model reference to pin its own thinking level independent of its tier, e.g. a model that only performs well at its own maximum setting can still live under `high` (so moderately-hard tasks reach it and it takes part in escalation normally) while always running at `max`:
+
+```json
+"high": {
+  "models": [{ "provider": "opencode-go", "id": "kimi-k3", "effort": "max" }]
+}
+```
+
+Omit `effort` and a model just uses its tier's own name, as before.
+
+On every turn, Auto asks the `medium` tier's first healthy model (the "default model") to classify the turn as `minimal`, `low`, `medium`, `high`, `xhigh`, or `max`, then routes to the resolved tier: if the classified level has no configured models, it steps toward `medium` until it finds one (a classified `low` with nothing configured there falls back to `medium`). Within that tier it picks the first model that isn't in a failure/rate-limit cooldown; if every model in the tier is unhealthy, it escalates to the next *higher* configured tier; if nothing anywhere is healthy, it uses the first configured model anyway rather than blocking the turn, with a warning.
+
+`/model` shows a separate entry per configured tier — "Auto (auto)" for the classify-every-turn behavior above, plus "Auto (medium)", "Auto (high)", and so on for each tier that has at least one model configured (tiers with nothing configured don't get an entry). Picking a specific one pins Auto to that tier: every turn skips classification and routes directly within it — still with the same failover, escalation, and health tracking as the adaptive mode, just without asking a model to judge complexity first. This list is fixed at startup from whatever's configured then, so adding a new tier to `autoRouter` needs a Pi restart before its "Auto (\<tier\>)" entry shows up.
+
+Health is tracked from two sources. Router-observed traffic (HTTP status codes, rate-limit headers, and message-level provider errors that never surface as a bad HTTP status) sets an immediate cooldown the moment any turn against a configured model fails — whether Auto routed there itself or you picked it manually from `/model`; a model configured in `autoRouter` is tracked the same way either way. Separately, best-effort real quota reconciliation runs at session start and on `/usage`, for providers with a known quota source: Anthropic, OpenAI Codex, Z.ai, Kimi Coding, and OpenCode Go via their HTTP APIs (using the same credentials Pi already has for each), plus Minimax via its `mmx` CLI (`mmx auth login`) since MiniMax has no HTTP quota endpoint of its own. This is what lets the router self-correct for usage consumed truly outside its view (a different session or machine, another tool, or before Auto was set up) instead of only reacting to its own observations. Codex specifically reports quota per-model for models it meters individually (its own `additional_rate_limits` entries) — those are independent of its account-wide limit in both directions, so a model with its own entry is neither blocked by, nor shielded by, the account-wide state; only models without one fall back to it. Providers without a known quota source simply stay on router-observed data.
+
+Run `/usage` to see health and usage for every configured model, grouped by tier. Each row shows its cooldown status if any, the real "verified usage" reported by the provider's own quota API when available — always normalized to "X% used" regardless of how the provider itself reports it, with each window labeled by its real duration rather than a vague placeholder where the provider's response makes that derivable (e.g. "7d 5% used", "5h 16% used, weekly 11% used") — and separately the request/token/cost totals *this Pi installation* has observed for that model, whether Auto routed there or it was picked manually. The latter still won't reflect usage from other sessions/machines/tools or from before Auto started tracking, which is exactly what verified usage is for. Shown as a bordered dashboard in the TUI, or a compact summary elsewhere (including Pi Web).
+
+`/usage` also shows the last several routing decisions under "Recent classifications" — what the classifier's raw reply actually was, the level it parsed to, and the tier/model it routed to. The classification call itself is otherwise a throwaway completion whose result would normally vanish the moment it's parsed, so if a turn ever looks under- or over-routed, this is what to check first rather than guessing from the code.
+
+The `/model` picker's effort/thinking control is inert while any Auto entry is selected, since effort is chosen per turn (or fixed to the pinned tier) internally. `/model` keeps showing whichever Auto entry you picked selected even after routing: the real model is only swapped in for the duration of each turn and swapped back to that same inert Auto placeholder as soon as it settles, so reopening `/model` between turns still shows "Auto (auto)" or "Auto (high)" (whichever you picked), not whichever model last handled a turn. A `🔀 Auto (<effort>)` badge in the TUI footer tracks the most recently applied thinking level regardless of which Auto entry is currently selected. Manually picking a real (non-Auto) model from `/model` turns Auto off; reselecting any Auto entry turns it back on.
+
+If you've scoped `/model` with `enabledModels` (or `--models`), Pi's picker defaults to showing only that scoped list, hiding everything else — including every Auto entry — behind a manual Tab to "all". At session start, Auto best-effort appends an `auto/*` pattern to `enabledModels` (only when scoping is already configured, and only if it isn't already present) so every Auto entry shows up in the default scoped view too, without changing anything else about what's scoped.
+
+### Requirements
+
+- Pi 0.84.1
+- Network access from the machine running Pi, for the optional quota reconciliation calls (never required — routing and `/usage` work fully offline from router-observed data alone)
+- For Minimax quota reconciliation specifically: MiniMax's own `mmx` CLI on `PATH`, logged in via `mmx auth login`. Without it, Minimax models just stay on router-observed data like any other unsupported provider.
+
 ## Worktrees
 
 Run `/worktree <name>` to create `<repo-root>/.pi/worktrees/<name>`, run the optional `.pi/worktrees/setup.sh`, and move the active conversation into a replacement session rooted in the managed checkout. The backward-compatible default creates or reuses local branch `<name>`; a missing branch starts at the selected checkout's `HEAD`.
@@ -129,5 +188,5 @@ bun install --frozen-lockfile
 bun run check
 bun test
 bun run webBuild
-pi -e ./extensions/session-footer.ts -e ./extensions/pr-footer.ts -e ./extensions/subagents.ts -e ./extensions/worktree.ts -e ./extensions/web-sessions.ts
+pi -e ./extensions/session-footer.ts -e ./extensions/pr-footer.ts -e ./extensions/subagents.ts -e ./extensions/worktree.ts -e ./extensions/web-sessions.ts -e ./extensions/auto-router.ts
 ```

extensions/auto-router-classify.ts

@@ -0,0 +1,119 @@
+import type { Api, Model } from "@earendil-works/pi-ai";
+import { uuidv7 } from "@earendil-works/pi-ai";
+import type { ModelRegistry } from "@earendil-works/pi-coding-agent";
+import type { AutoRouterEffortLevel } from "./auto-router-settings.js";
+
+const CLASSIFY_TIMEOUT_MS = 15_000;
+const VALID_LEVELS: readonly AutoRouterEffortLevel[] = [
+  "minimal",
+  "low",
+  "medium",
+  "high",
+  "xhigh",
+  "max",
+];
+const DEFAULT_LEVEL: AutoRouterEffortLevel = "medium";
+
+const SYSTEM_PROMPT = `You triage the complexity of a single upcoming coding-agent turn so it can be routed to an appropriately capable model. Reply with exactly one word, lowercase, no punctuation: minimal, low, medium, high, xhigh, or max.
+
+- minimal: rote, no real reasoning needed. A one-word answer, a pure formatting pass, a trivial rename, echoing back something already known.
+- low: simple and mechanical, but not entirely rote. One-line edits, small lookups, answering a quick factual question about the codebase.
+- medium: a typical coding task. Implementing a small-to-moderate feature, fixing a well-understood bug, writing straightforward tests. This is the default for ordinary work.
+- high: meaningfully harder. Multi-file refactors, tricky or intermittent bugs, non-obvious architectural changes, tasks that require holding a lot of context at once.
+- xhigh: very hard, high-stakes, or open-ended. Large-scope redesigns, subtle correctness/security-critical work, or reasoning-heavy problems where getting it wrong is costly.
+- max: the hardest, rarest cases. Deep multi-step reasoning under real stakes — major system redesigns, subtle distributed-systems or security bugs, decisions with significant real-world consequences.
+
+Reply with only the single word.`;
+
+export type ClassificationUsage = {
+  input: number;
+  output: number;
+  cost: number;
+};
+
+export type ClassificationResult = {
+  level: AutoRouterEffortLevel;
+  usage?: ClassificationUsage;
+  /**
+   * The classifier's raw reply (trimmed/lowercased), or a bracketed placeholder when the call
+   * itself failed. Callers should log this alongside `level` - otherwise there's no way to tell
+   * apart "the model genuinely said medium" from "parsing picked the wrong word out of a messy
+   * reply" after the fact, since the model call itself is never persisted anywhere else.
+   */
+  reply: string;
+};
+
+function numeric(value: unknown): number {
+  return typeof value === "number" && Number.isFinite(value) ? value : 0;
+}
+
+/**
+ * Classify a turn's complexity using the given (default/medium-tier) model. Never throws and
+ * never blocks indefinitely: a bounded timeout, a provider error, or an unparseable reply all
+ * fall back to `medium` so classification can never stall or break the user's turn.
+ */
+export async function classifyTurnComplexity(
+  modelRegistry: ModelRegistry,
+  model: Model<Api>,
+  prompt: string,
+  hasImages: boolean,
+): Promise<ClassificationResult> {
+  const controller = new AbortController();
+  const timeout = setTimeout(() => controller.abort(), CLASSIFY_TIMEOUT_MS);
+  try {
+    const text = hasImages
+      ? `${prompt}\n\n(This turn also includes attached images.)`
+      : prompt;
+    const response = await modelRegistry.complete(
+      model,
+      {
+        systemPrompt: SYSTEM_PROMPT,
+        messages: [
+          {
+            role: "user",
+            content: [{ type: "text", text }],
+            timestamp: Date.now(),
+          },
+        ],
+      },
+      {
+        signal: controller.signal,
+        reasoningEffort: "off",
+        cacheRetention: "none",
+        sessionId: uuidv7(),
+        maxTokens: 20,
+      },
+    );
+    const reply = response.content
+      .filter(
+        (block): block is { type: "text"; text: string } =>
+          block.type === "text",
+      )
+      .map((block) => block.text)
+      .join("")
+      .trim()
+      .toLowerCase();
+    // Match whichever valid level word appears *first in the reply text*, not the first one
+    // in VALID_LEVELS' own order - a naive `VALID_LEVELS.find(word-boundary test)` would let an
+    // earlier-in-that-list word like "medium" win over a later one like "high" even when "high"
+    // is the word the model actually led with (e.g. "high complexity, more than a medium task"),
+    // silently downgrading the classification. `\b(...)\b` as one alternation also keeps the
+    // existing "high" vs "xhigh" substring safety: `\b` can't match between two word characters,
+    // so "high" never matches inside "xhigh" regardless of alternation order.
+    const match = reply.match(new RegExp(`\\b(${VALID_LEVELS.join("|")})\\b`));
+    const level = (match?.[1] as AutoRouterEffortLevel | undefined) ?? DEFAULT_LEVEL;
+    const usage = response.usage
+      ? {
+          input: numeric(response.usage.input),
+          output: numeric(response.usage.output),
+          cost: numeric(response.usage.cost?.total),
+        }
+      : undefined;
+    return { level, usage, reply: reply || "(empty reply)" };
+  } catch (error) {
+    const message = error instanceof Error ? error.message : String(error);
+    return { level: DEFAULT_LEVEL, reply: `(classification failed: ${message})` };
+  } finally {
+    clearTimeout(timeout);
+  }
+}