fix(llm): provider-availability failures no longer block fallover

[plombeer31]

What

A failure meaning "this provider link is unusable" must not be filed under a category that blocks fallover. Two such failures were.

(a) llama-server 4xx. classifyFailure mapped every LlamaServerError with a non-null status under 500 to "grammar". So an HTTP 404 — the configured localModels.url does not expose the completion endpoint (wrong URL, or a server that is not a llama-server) — and a 405 (wrong method) were reported as grammar failures.

(b) subscription-CLI providers. SubscriptionCliNotInstalledError and SubscriptionCliAuthError are plain Errors, so classifyFailure fell through to its catch-all return "tool".

(c) a duplicate copy of the same taxonomy in step-executor.ts. toLlmFailure carried its own hardcoded status === null || status >= 500 ⇒ transport, else grammar split. executeStep wraps every escaping error through toLlmFailure and rethrows that wrapper, and classifyFailure's first line short-circuits on err instanceof LlmFailure — so on the path that produces the message the user reads, the duplicate decided the category and (a) never applied. This was found in review of the first commit here and is fixed in 33af1f6; without it the llama half of this PR would have changed only the fallover decision, not the chat message or the hint.

Three consequences, all in-repo:

1. The user reads Turn failed [grammar]: llama-server returned http 404, or Turn failed [tool]: "claude" was not found on PATH… — a tool bug it is not. Both are unactionable.
2. shouldAdvance() returns {advance:false} for grammar and tool, so the fallback chain refuses to move to the next provider even though the current link is permanently broken. Its own doc comment states the opposite intent for the cloud path: every OpenAiHttpError classifies as transport regardless of status, "so a 404 model-not-found or a 401 dead key advances too — a different link may have the model or a working key." The local and CLI paths contradicted that. A configured claude/codex provider that is simply not installed blocked the chain permanently.
3. format-agent-error-for-chat.ts appends formatLlamaUnreachableHint(local.llamaUrl) only when category === "transport". The one case where "check your llama URL" is exactly the right advice — a 404 from a wrong URL — was the one case that never got the hint.

Sentry:

• CLI-B7 — 141 events / 9 users, first seen 2026-08-28, still firing. category=grammar, cause_type=LlamaServerError, http_status = 400 (118), 404 (15), 405 (8), release = 0.5.4 (111), 0.4.2 (28), 0.5.3 (2).
• CLI-BE — 9 events / 6 users, win32. http_status = 404 (6), 400 (3), same grammar category, releases 0.4.2 / 0.5.4 / 0.5.1.
• CLI-BH — 7 events / 5 users, first seen 2026-08-29. error_type=ToolExecutionError, category=tool, cause_type=SubscriptionCliNotInstalledError, releases 0.5.1 / 0.4.2.

The grammar + LlamaServerError signature in CLI-B7 / CLI-BE is precisely new GrammarError(err.message, "", { cause: err }) in toLlmFailure — i.e. those two clusters are addressed by 33af1f6, not by the classifier change alone. The 404/405 slice of them (29 of 150 events) stops being reported as grammar; the 400 slice (121 events) is deliberately left where it is, see Deliberately out of scope. CLI-BH goes through the catch-all arm, which already defers to classifyFailure, so it is fixed by the classifier change on its own.

How

src/llm/reliability/classify-failure.ts:

• The LlamaServerError 4xx arm is split against a named set. Statuses that describe the endpoint / auth / availability rather than the request return "transport": 401, 402, 403, 404, 405, 408, 409, 429 (joining the existing null and >= 500). None of these repeat identically on a different provider.
• Statuses that mean this request was rejected as malformed or oversized stay "grammar": 400, 413, 422, and — deliberately conservative — any other 4xx not named above. The default stays on the non-advancing side so an unfamiliar status cannot silently start burning the chain.
• SubscriptionCliNotInstalledError and SubscriptionCliAuthError route to "transport". SubscriptionCliInvocationError deliberately does not: the binary ran and came back unhappy, which is not evidence the link is unusable.

Imported straight from ../provider/subscription-cli/subscription-cli-errors.js, not the subscription-cli/index.js barrel. subscription-cli-errors.ts has zero imports of its own, so there is no cycle; the barrel would have dragged the providers and adapters into the reliability layer. Same shape as the existing ../provider/openai/openai-http.js import.

src/agent/step-executor.ts: the LlamaServerError arm of toLlmFailure no longer restates the split — it asks classifyFailure(err) and maps transport to TransportError(message, status, url), anything else to the historical GrammarError(message, ""). The cause chain is untouched in both arms, so the scrubber's causeType still resolves. classifyFailure cannot answer cancelled/model/tool for a LlamaServerError (its own arm returns only transport or grammar, ahead of the abort and network branches), and an aborted step is already claimed by the ctx.signal.aborted check above this arm, so no other category is laundered into a TransportError. This was the last hardcoded llama status split in src/; llama-server-client.ts (isRetryableLlamaError) and should-advance.ts (isImmediateSignal) read statuses for different decisions and are unchanged.

Doc comments updated to the new rule in classify-failure.ts, failure-category.ts (its grammar: bullet said "also covers HTTP 4xx from llama-server") and should-advance.ts (its grammar bullet said "A grammar/4xx failure is request-shape").

isImmediateSignal in should-advance.ts needed no change and does not double-count: it reads the status straight off LlamaServerError, so a 429 or 408 that is now transport is correctly immediate, while 401/402/403/404/405/409 are not, and the CLI errors carry no status field. To be precise about what that buys — ProviderFallbackChain.advanceFrom returns the next link on the first fallover-worthy failure whether or not it is immediate; immediate only arms the breaker cooldown straight away instead of waiting for failureThreshold consecutive failures, i.e. it governs how long the failed link stays quarantined across later turns, not the in-turn switch. The AdvanceDecision doc said otherwise and has been corrected.

Trade-offs accepted

A subscription CLI can be called signed-out by its own output. mapCliFailure regex-matches stderr + "\n" + stdout, and stdout carries the CLI's own model output. So a non-zero exit whose text merely discusses auth — "unauthorized", "not logged in" — is relabelled SubscriptionCliAuthError. Before this PR that misfire stopped the chain and printed "not signed in, run /login", which at least named itself. After it, the chain falls over silently, and a genuine sign-out on a multi-link chain becomes a silent, repeated degradation to a paid cloud provider. On balance still the right trade — a CLI that cannot serve should not pin the chain — but it is a real behaviour change and it is undocumented anywhere else. The regex is a pre-existing weakness and is deliberately not touched here.

403 and the proxy case. A llama-compatible endpoint behind LiteLLM / vLLM / a corporate gateway can answer 403 for a content rejection, which would fail identically on every link — so the chain burns every provider before giving up. That is consistent with what the cloud path already does for every OpenAiHttpError status, and src/llm/llama-server-auth-probe.ts:31 already treats a llama 401/403 as an auth verdict, so 403 is placed with the existing precedent rather than against it; 402 and 409 are almost certainly unreachable from llama.cpp itself and are set members only for the same proxied deployments.

Tests

• classify-failure.test.ts: a table over statuses — 400/413/422 → grammar, 401/402/403/404/405/408/409/429 → transport, 500/503 → transport, null → transport, plus an unlisted 418 → grammar guard for the conservative default; the two subscription-CLI errors → transport, and SubscriptionCliInvocationError still → tool.
• should-advance.test.ts: llama 404 and 405 advance without arming the breaker; llama 429 and 408 advance and arm it on the first failure; llama 400 still does not advance; both subscription-CLI errors advance.
• format-agent-error-for-chat.test.ts: drives the whole production path rather than imitating it — AgentLoop runs a step whose llmComplete throws a raw LlamaServerError, executeStep normalises it through toLlmFailure, the loop's catch classifies that wrapper and emits loop_failed { category, error }, and the assertion formats exactly those fields the way agent-event-reducer's loop_failed case does. A llama 404 and 405 now carry the unreachable hint; a llama 400 still renders as a bare grammar failure.

Not vacuous.

• With classify-failure.ts reverted and the tests kept, 18 of the new tests fail (10 in classify-failure.test.ts, 6 in should-advance.test.ts, 2 in format-agent-error-for-chat.test.ts). The remaining new rows are regression guards for behaviour that is intentionally unchanged.
• With only the toLlmFailure change (33af1f6) reverted, the two format-agent-error-for-chat.test.ts pipeline rows fail with Turn failed [grammar]: llama-server returned http 404 / … 405 — the exact production symptom. The earlier, hand-composed version of that test passed in that state, which is why it was rewritten.

Verification

• npm run lint (tsc --noEmit): clean.
• npx vitest run src/llm/reliability src/llm/fallback src/tui/format-agent-error-for-chat.test.ts src/llm/provider/subscription-cli — 18 files, 197 tests, all passing.
• npx vitest run src/agent (the suite over the changed step-executor.ts) — 15 files, 247 tests, all passing.
• npx vitest run agent-loop step-executor error-scrubber llama-server-provider llama-server-client llm-fallback-seam replay-session tasks tracing turn-controller tui/llm-panel — 44 files, 483 tests, all passing.
• Full suite not gated on (known flaky); no failure introduced by this change.

Deliberately out of scope

A llama-server 400 is by far the largest bucket in CLI-B7 (118 of 141) and a good share of those are almost certainly context-overflow — the prompt exceeded the server's n_ctx. Sniffing the 400 response body to separate "prompt too long" from "malformed request" is not attempted here. It needs a taxonomy change, not a status remap: ModelFailureReason has no context value, and the correct handling for an overflow (compact or trim, then retry the same provider) is neither grammar nor transport. Both currently land on the non-advancing side, which is the safer of the two answers, so this change leaves 400 exactly where it was.