fix(chat): stop context overflows from leaving sessions stuck

[mudler-agent]

Sessions on regolo (a LiteLLM proxy in front of vLLM) could overflow the context and then stay stuck: every turn, and a manual /compact, failed the same way. This PR fixes the cause and makes every stage of overflow recovery able to make progress.

What went wrong

Reproduced against the live endpoint:

• regolo's /model/info reports max_input_tokens: 96000 and max_output_tokens: 96000. The vLLM behind it accepts 210000. LiteLLM does not enforce 96000.
• nib sent max_completion_tokens=96000 on every request, including the compaction summary. vLLM rejects a request when prompt + requested output exceeds 210000, so any prompt above ~114k failed. nib's budget only reserved 4096 tokens for output.
• On the first overflow nib learned the 210000 window from the error, so compaction fired at ~165k, above the ~114k that already failed.
• The summary request also reserved 96000 output tokens, so it overflowed too. Its retry scaled the prompt by window/total, which cannot fit a fixed reservation. Compaction failed, and the failed-turn rollback restored the history from before recovery compacted, so the next turn overflowed again.

Changes

Root causes:

• Clamp max_tokens per request to the room the window has left (clampOutputTokens in the tracked LLM wrapper, tool schemas counted).
• Summary that always fits. The summary has its own output cap (compaction.summary_max_tokens, default 16384). An overflowing summary shrinks by the overshoot the backend reports, and the compaction boundary moves earlier. Messages cut from the summary stay verbatim.
• Rolling summary. When one call cannot hold the head, it is summarized in chunks, each merged with the running summary, so the result is small and nothing is dropped. When the window is known and the head is already too large, rolling starts at once instead of truncating the summary input.

Recovery:

• Overflow error catalog. Detection is one ordered list of regex rows with named groups (window, total, input, output), backed by fixtures in chat/testdata/overflow/ (the regolo ones are live captures). Users can add rows with compaction.overflow_patterns. The HTTP status gates weak evidence only; LocalAI reports an overflow as HTTP 500, so provider rows classify at any status.
• Three overflow kinds. KindContext compacts. KindBudget (prompt fits, prompt + output does not) retries with a lower max_tokens and never compacts. KindOutputCap lowers the output cap.
• Escalation chain. Compaction, then a smaller kept tail, then a forced prune, then hard truncation, then a non-LLM fallback compaction. Each step is validated (validateCompaction: size, budget, tool-call pairing). The whole chain counts as one recovery.
• The failed-turn rollback keeps the compaction. It drops only the failed turn's messages.

Broken tool calls (with mudler/cogito#80):

• Interrupted streams and invalid tool arguments get their own retry classes. An interrupted stream is retried at most 3 times, invalid arguments at most once, instead of the 10 turn retries × 3 cogito attempts an unknown error used to get. Each has a clear message instead of unexpected end of JSON input.
• A tool call cut off because the context window ran out is context exhaustion. This is detected from the usage cogito now reports (prompt + completion ≥ 95% of the window, as llama.cpp/LocalAI stop at n_ctx) or from the clamp having lowered max_tokens.
  • If the prompt is at least 25% of the window, nib compacts, then retries once with a one-turn note telling the model which call was cut and to split it.
  • Otherwise it retries once without compacting, lowering the reasoning effort for that turn when reasoning filled the window.
  • The note is never stored in the history.
• No per-request output ceiling. It would cut legitimate long tool calls such as writing a whole file.

Also:

• Progressive, sticky tool-output compression in four levels. Levels only escalate, so the backend's prefix cache stays warm.
• Tool-schema floor, measured. The request wrapper measures what each request really carries in tool schemas and system messages, MCP servers and cogito's own tools included, and scales it by the tokenizer ratio the backend reports (clamped to 1–4×; image requests do not calibrate). The floor is deducted from the validation budget.
• Per-MCP-server attribution. When the floor crosses the warning threshold, the user sees one notice, for example: tool schemas use 81k of the 96k context window — github 34k (52 tools), jira 22k (31 tools)…: disable MCP servers or filter their tools.
• No compaction when the schemas are the problem. If the floor alone leaves no room for the smallest possible request, recovery and auto-compaction do not touch the history. The turn ends with that same message, and the conversation is kept.
• Bump cogito to 5aa9d6d: typed status errors from the streaming path (fix(clients): return a typed status error from the LocalAI streaming path cogito#79), and sentinel errors for interrupted streams and broken tool-call arguments (fix(tools): stop retrying broken tool-call arguments with identical requests cogito#80). The old text forms are still parsed.

Testing

• go test ./... -count=1 (with an isolated HOME, see below), go vet ./... and go test -race ./chat/ -count=1 pass.
• An HTTP regression test models the regolo setup at 1:10 scale (LiteLLM window 9600 with a 9600 output cap, vLLM enforcing 21000 on prompt + max_tokens). After the one turn that teaches nib the real window, 12 more turns succeed while the prompt grows past the old failure point, and a manual compaction succeeds.
• Each commit was written test-first and reviewed separately.

Notes for review

• Output clamping applies to OpenAI-compatible clients, where the session knows the output cap. Other adapters are unchanged.
• A generic overflow phrase in a 5xx response is now retried as transient instead of treated as fatal.
• Before the first request of a session the floor is an estimate (built-ins plus each MCP server's listed tools). It is only used to block recovery when the estimate alone is over the window.
• The per-server listing is cached and refreshed when the set of MCP servers changes.
• Follow-up, not in this PR: deferred MCP tool loading (full schemas only for built-ins and a bounded set of MCP tools, the rest by name with a load-on-demand tool). It is the structural fix for large MCP setups.
• Test isolation (existing bug, also on master): cmd tests such as TestRunCLIWritesTheSummaryToStderrOnly read ~/.config/nib and send their chat request to the saved real provider instead of the test server. On a machine with a saved /login they make billed requests and time out under rate limits. Run the suite with an isolated HOME until that is fixed separately.
• The implementation plan is docs/specs/2026-09-27-context-management-robustness.md (not committed).

🤖 Generated with Claude Code