Repository navigation
Conversation
The reasoning steps of WithForceReasoning (before the tool pick and before parameter generation) send no output cap of their own, so a model that never closes the reasoning argument runs to the client's global cap (16384 by default), and the length retry then doubles it. Observed with gemma4-26b-a4b behind LocalAI: one forced reasoning call streamed 12,969 tool-call deltas for 478-501 s until the cap, and the turn never answered. The reasoning steps now carry max_tokens from WithReasoningMaxTokens (default DefaultReasoningMaxTokens = 2048; 0 uses the default, a negative value keeps the previous uncapped behaviour). The cap goes on the request, where the bundled clients take it over their own default (see SetMaxTokens). A finish_reason=length on such a capped step is final: no length retry, the call returns an error wrapping errOutputCapReached; a complete tool call that is cut at the cap is still used. The parameter-reasoning step already falls back on any error; the tool-pick step now continues without the reasoning on this error only, other errors stay fatal. A reasoning that finishes normally is used as before. The option is forwarded to plan sub-executions. Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de> Assisted-by: Claude Opus 5.5 <noreply@anthropic.com>
With WithForceReasoning, a picked sink state that has parameters (an embedder's own sink such as LocalAGI's no_tool_to_call) gets its arguments from a forced parameter-generation request that carried no output cap, so it ran to the client's global cap (16384 by default). Observed with the daily-driver model behind LocalAI: the model answered inside the sink's "reasoning" argument and then repeated "17 mal 23 ist 391." until the cap (argumentsLen=46865). The backend reported finish_reason=tool_calls for the cut call, so the arguments correction re-ran the same request up to maxRetries times, and the turn hung for more than 420 s. With finish_reason=length the length retry would have doubled the cap instead. The sink's parameter step now carries max_tokens from WithSinkStateMaxTokens (default DefaultSinkStateMaxTokens = 2048; 0 uses the default, a negative value keeps the previous uncapped behaviour). Only the sink is capped: real tools keep the client's cap, since their arguments may legitimately be long. A capped step that spends its whole cap counts as cut even when the backend reports finish_reason=tool_calls: a complete call is still used, otherwise the step fails with errOutputCapReached and neither the length retry nor the arguments correction runs. This also applies to the reasoning steps capped by WithReasoningMaxTokens, which share the capped decision. On a cut, the parameter generation keeps its existing fallback (the arguments from the pick). The sink is not executed and the turn's answer comes from the closing completion after the loop, so nothing the user sees is lost and the turn ends. The option is forwarded to plan sub-executions. Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de> Assisted-by: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de> Assisted-by: Claude Opus 5.5 <noreply@anthropic.com>
walcz-de
added a commit
to walcz-de/LocalAI
that referenced
this pull request
Oct 3, 2026
cogito fcb7c3a: the sink state's parameter step gets its own output cap (WithSinkStateMaxTokens, default 2048) — mudler/cogito#85, stacked on mudler#84. A LocalAGI agent repeated its answer inside the no_tool_to_call argument (46,865 bytes) and hung for more than 7 minutes; argument correction retried each truncated call at 16K tokens. Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de> Assisted-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #84. This branch builds on
upstream-pr/cap-reasoning-stepand reuses its helpers (decisionWithStreamingCapped,errOutputCapReached). The new commits are the last two: the fix and its README note. Once #84 is merged, a rebase leaves only those.With
WithForceReasoning, the sink state's arguments are generated in their own request (generateToolParameters), which forces the sink tool and carries no output cap. A model that keeps repeating its answer inside thereasoningargument runs to the client's global cap (16384 by default).Behind LocalAI the truncated call came back with
finish_reason=tool_callsrather thanlength, so the argument-correction path retried the whole request, each time up to 16K tokens. With a LocalAGI agent (sinkno_tool_to_call) one turn produced a 46,865-byte argument repeating the same sentence and hung for more than 7 minutes. The sink value is not used for the answer, which comes from the final ask.This PR adds
WithSinkStateMaxTokensand sends it asmax_tokenson the sink's parameter step only:DefaultSinkStateMaxTokens = 20480uses the defaultA capped step that uses up its whole budget (
CompletionTokens >= cap) counts as truncated even when the backend reportstool_calls. Such a step gets neither a length retry nor argument correction. The loop takes its existing fallback and answers as usual. A complete call within the cap is used as before.The same completion-token check also applies to the reasoning steps from #84, because they share the helper. The option is forwarded to plan sub-executions, and the README's sink state section documents it.
Tests (
sink_cap_test.go, 7 specs, streaming and non-streaming mock LLM):lengthand withtool_calls→ exactly one request carryingmax_tokens=2048, and the answer still arrives;write_document) with a 6,144-delta argument → no cap, full argument;Without the fix all 7 fail. The full suite with
--label-filter='!e2e'passes.🤖 Generated with Claude Code
https://claude.ai/code/session_01Y8ngj32vPUcLs3QKgTYCxy