Repository navigation
fix(clients): return a typed status error from the LocalAI streaming path - #79
Merged
Merged
Conversation
…path A non-200 answer to a streaming request came back as plain text, "localai stream: status 400: <body>", so a caller could read the HTTP status only by parsing it out of the message. The non-streaming path returned the decoded *openai.APIError without its status: go-openai tags HTTPStatusCode json:"-", so decoding leaves it zero. Both paths now share statusError. A body that holds an OpenAI error object becomes that *openai.APIError with HTTPStatusCode and HTTPStatus set. Any other body becomes an *openai.RequestError that keeps the raw body. The streaming error keeps its "localai stream: " prefix and wraps the typed error, so errors.As reaches it. Callers need the status to classify a failure: a context overflow is a 400 or a 413 on most backends, and a 500 whose body mentions "context" is not one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mudler
added a commit
to mudler/nib
that referenced
this pull request
Sep 26, 2026
* fix(chat): clamp max_tokens per request to the room the window has left The client sent its full output cap as max_tokens on every request. vLLM rejects prompt + max_tokens > window, so with a cap that is a large part of the window every long prompt, compaction summaries included, overflowed. The turn's LLM wrapper now sets max_tokens to min(cap, window - prompt estimate - 512), floored at 1024, when the request carries no explicit value. The session records the cap the factory and discovery resolved for the current model. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(chat): fit the compaction summary by moving the boundary, not truncating A summary that overflowed was retried once, at a size scaled by window/request. The output the request reserves does not scale with the prompt, so with a large reservation the retry overflowed too, and the session stayed stuck. The summary request now reserves min(output cap, summary_max_tokens, default 16384) through CreateChatCompletion, and its prompt is fitted into the window minus that reservation. On an overflow, summarizeFitting subtracts the stated overshoot from the prompt and moves the boundary earlier until the head fits, up to 6 attempts. Messages that leave the head stay in the history verbatim. End-of-turn, mid-turn and /compact all use this loop, and the artifact holds the head that was summarized. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(chat): summarize an overflowing head in rolling chunks When the summary of the whole head overflowed, the boundary moved earlier and the cut messages stayed verbatim. Nothing was lost, but a far move left the history barely smaller, so the next request compacted again. Now the head is summarized in chunks from oldest to newest, each fitted to the retry target. Every chunk after the first carries the running summary and asks to merge it. A chunk that overflows anyway shrinks and passes the rest to the next chunk. Chunks end only on tool-call boundaries, and an over-large message is cut down, not skipped. After 8 chunks the uncovered messages stay verbatim, as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(chat): validate a compaction result before it is sent Add validateCompaction, which rejects an empty result, a result that grew, a no-op, a result over the budget, and broken tool-call pairing. The sentinel errors are exported so overflow recovery can switch on them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(chat): estimate the context the tool schemas and system prompt take SchemaBudget measures the fixed per-request overhead (tool schemas plus system prompt, bytes/4) against the context budget and window, and logs a warning when it passes 60% of the budget or 90% of the window. toolOptions records the tool definitions it registers so the session can measure them; cogito keeps its own list private. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(chat): trim the history step by step when it overflows iterativeTrim runs compaction, compaction with a smaller tail, an aggressive tool-output prune and a hard truncation in turn, and validates each result with validateCompaction before it moves on. compactHistory now calls compactHistoryKeep, so a smaller tail does not write the shared config. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(chat): compact the history without an LLM as a last resort Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(chat): compress old tool output in sticky, graduated levels Add compressToolOutput with four levels (full, truncated, outline, elided) and Session.progressivePrune, which runs pruneMessages and then assigns levels by position and context pressure. The turn's messages manipulator now calls progressivePrune, so every request carries the levels. Levels are recorded by tool_call_id with their rendered text, only go up, and are assigned in one batch only when pressure rises into a higher band. The prompt prefix changes at most once per threshold crossing. Compaction and the forced prune of the overflow recovery render the same levels, so they see what the requests sent. An elided result with no artifact names the tool that produced it. The mid-turn compaction tests now use 7500-token results. At 9000 the request passed 90% of the budget, the older result was elided and the request fell under the trigger, so no compaction ran to test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(chat): classify overflow errors from a catalog of patterns Replace the substring markers and the "first two N tokens" guesser with an ordered list of regex rows. Named groups (window, total, input, output) say what each figure means. classifyOverflow applies the structured error code, the rows, an output-cap guard and a 413 rule, and returns one of KindContext, KindBudget or KindOutputCap. Fixtures in chat/testdata/overflow include the two regolo captures on both cogito paths, and the typed streaming text form cogito is moving to. Each row is marked strong or weak. The status gate (400, 413 or unknown) applies only to weak evidence: the generic phrase rows and the 413 rule. A structured code, a provider row or a config row classifies at any status, because LocalAI and llama.cpp report an overflow as HTTP 500 (gRPC Internal). isContextOverflow is now KindContext only. isBudgetOverflow and isOutputCapError are added for the recovery work. learnedWindowFrom, overflowFigures and overflowDetail read the catalog. compaction.overflow_patterns adds user rows ahead of the built-ins. Changes to existing tests and callers: - summary_shift_test: TestSummaryGivesUpAfterMaxAttempts expects isWindowOverflow. Its vLLM rejections state an input that fits, so they are budget overflows now. - config_test: compare Compaction with reflect.DeepEqual, because the struct now holds a slice. - The summary retry (summarizeFitting, rollingCover) shrinks on context and budget overflows through canShrinkSummary. - classifyBackendError reads classifyOverflow: a LocalAI overflow at HTTP 500 stays fatal; a generic phrase in a 5xx is retried. - No test learned a window from a wording that no row names. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(chat): recover from overflow with trim, fallback and kind-specific retries Overflow recovery in SendMessage now acts on the overflow kind: - context: run iterativeTrim, then basicCompact when trimming cannot make the history fit. The whole chain is one recovery attempt, and the retry is announced only when a step changed the history. - budget: do not compact. Retry once with the output reservation set to window - input - outputSafetyMargin, from the backend's figures, through a one-turn override that requestLimits reads. When that is below minOutputTokens, or the lowered retry overflows again, only a smaller prompt helps, so the turn goes through the context chain. - output_cap: do not compact. Lower the session's output cap (and the client's) to the stated maximum, never raising it, and retry once. When the retry after recovery still overflows, the rollback keeps the compaction and removes only the failed turn's own messages. Before, it restored the pre-turn history, so the next turn overflowed again and the session was stuck. Changes to existing tests: the overflow fixtures seeded a history of a few bytes, which a summary cannot shrink, so the validated chain rejected it. They now seed longer replies. The tests that need the whole chain to fail keep the tiny history (tinyHistory). The regolo regression test runs over HTTP at 1:10 scale: vLLM window 21000, LiteLLM /model/info 9600/9600. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * chore(deps): bump cogito to return typed status errors from streams cogito 01190bf (mudler/cogito#79) returns a non-200 streaming answer as an *openai.APIError or *openai.RequestError that carries the HTTP status, instead of the text "localai stream: status 400: <body>". errorStatus now reads the status from the typed error on the streaming path too; the text form stays supported. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(chat): roll a summary head known to be too large from the start With a known window, a head larger than the summary prompt budget still went out whole first, and fitSummaryInput cut long pieces and dropped middle messages to make it fit. That built a lossy summary input on purpose, and cost the same one call before rolling would start. Now such a head goes directly to rollingCover with the initial target (window less the summary output reservation, capped by ContextBudget). A head that fits, or an unknown window, keeps the single whole-head call and rolls only after an overflow. fitSummaryInput still cuts a single piece too large for a chunk on its own. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(chat): measure the tool-schema floor on the requests the turn sends SchemaBudget counted only the tools nib registers itself. cogito adds every MCP server's tools and its agent tools to each request, so with many MCP servers the real floor was far larger than the estimate and no warning fired. The request wrapper now records the size of the tools and system messages of each request that carries tools, raised to the backend's own overhead when the reported prompt tokens say more. SchemaBudget uses that floor once a request was sent, and the MCP servers' tools in the estimate before. It attributes the schemas per MCP server (cached until the server set changes), and the session reports the floor once through OnStatus when it crosses the warn threshold, and again at the error threshold. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(chat): keep the conversation when the tool-schema floor fills the window When the tool schemas and system prompt alone leave no room for the user's message and a minimal answer, no compaction makes a request fit. Overflow recovery compacted, trimmed and basic-compacted the history anyway, and then failed. Overflow recovery now checks the floor before the chain. When the floor plus the user's message and minOutputTokens is over the window, the turn ends with errSchemaFloor: the history is restored to the pre-turn state, the user's message stays on screen, and the message names the largest MCP servers, in the same words as the schema-budget notice. End-of-turn and mid-turn auto-compaction are skipped when the floor plus the last user message is over the context budget. A floor that is only an estimate blocks only when it alone is over the limit. TestSessionAutoCompacts reported 1000 prompt tokens against a 100-token window, which calibrates the floor far over the budget. It now reports 14000 against a 20000-token window: over the trigger, under the budget. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(chat): scale the schema floor by the tokenizer ratio The floor was raised to prompt_tokens minus the estimate of the non-system messages. That residual holds the tool schemas and also the tokenizer skew on the whole conversation, so the floor grew with the history: 5k of schemas under 60k of history on a backend counting 1.5x read as a 35k floor. With a large reserve, auto-compaction was skipped although the history was the problem, and the schema notice reported the skew as tool schemas. The wrapper now learns a ratio, prompt_tokens over the byte/4 estimate of the whole request, clamped to [1, maxTokenizerRatio], and the floor is the measured schemas and system prompt times that ratio. A request with an image part does not calibrate, as the estimate does not count images. The mid-turn trigger still uses the full residual. TestSessionAutoCompacts now runs against a backend that counts 1.5x the estimate, with a large history and half the window reserved. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(chat): classify interrupted streams and truncated tool calls cogito now names why a decision failed. An interrupted stream is retried at most 3 times and invalid tool arguments once, instead of the 10 turn retries every unknown error got. A tool call cut by finish_reason=length is a context problem when the window ran out: prompt plus completion reached 95% of it, or the output clamp lowered max_tokens for that request. nib then retries the turn once with a one-turn note that asks the model to split the call. It compacts first when the prompt is at least a quarter of the window, and lowers the reasoning effort for the turn when the reasoning filled it. The output cap is never lowered. When the cap was reached with room left, the turn fails with a message that names max_tokens. cogito's text for a truncated call says "context window", so the overflow catalog no longer reads it as a rejected request. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A non-200 answer to a streaming request came back as plain text (
localai stream: status 400: <body>), so a caller could read the HTTP status only by parsing the message. The non-streaming path returned the decoded*openai.APIErrorwithout its status: go-openai tagsHTTPStatusCodejson:"-", so decoding leaves it zero.Both paths now share
statusError:*openai.APIError, withHTTPStatusCodeandHTTPStatusset.*openai.RequestErrorthat keeps the raw body.localai stream:prefix and wraps the typed error, soerrors.Asreaches it.Why
nib classifies a failed request as a context overflow before it compacts the conversation. A context overflow is a 400 or a 413 on most backends, and a 500 whose body mentions "context" is not one, so nib has to gate on the status. With streaming (nib's default path) the status only existed as text.
This was found while debugging overflows on regolo (a LiteLLM proxy in front of vLLM), which answers with bodies like
{"error":{"message":"Requested token count exceeds the model's maximum context length of 210000 tokens. ...","type":"BadRequestError","code":"400"}}.Compatibility
localai stream: error, status code: 400, status: 400 Bad Request, message: <message>. The non-streaming*openai.APIErrortext gains the sameerror, status code: ...prefix now that its status is set.isContextOverflowError).status NNN:out of the text should switch toerrors.As.Tests
TestStreamStatusErrorIsTyped: a 400 with an OpenAI error object unwraps to*openai.APIErrorwith status 400 and the backend's message, and keeps the prefix.TestStreamStatusErrorWithoutErrorObject: a 413 with a plain body unwraps to*openai.RequestErrorwith status 413 and the raw body.TestCompletionAPIErrorCarriesStatus: the non-streaming*openai.APIErrorcarries status 400.All three failed before the change and pass after it.
make test: 74 passed, 0 failed, 1 flaked. The flake isagent_dispatcher_test.go:122, a timing-dependent dispatcher test outsideclients/; it failed on one run and passed on retry.🤖 Generated with Claude Code