Skip to content

fix(tools): bound the forced reasoning steps by their own output cap - #84

Open
walcz-de wants to merge 1 commit into
mudler:mainfrom
walcz-de:upstream-pr/cap-reasoning-step
Open

walcz-de wants to merge 1 commit into
mudler:mainfrom
walcz-de:upstream-pr/cap-reasoning-step

Conversation

@walcz-de

@walcz-de walcz-de commented Oct 2, 2026

Copy link
Copy Markdown

With WithForceReasoning, the forced reasoning calls before the tool pick and before parameter generation carry no output cap of their own. A model that never closes the reasoning argument therefore runs to the client's global cap (16384 by default), and the length retry then doubles it. With gemma4-26b-a4b behind LocalAI, one such call streamed ~13k tool-call deltas for ~8 minutes and the turn never answered.

This PR adds WithReasoningMaxTokens and sends it as max_tokens on both reasoning calls, so the request-level cap wins over the client default (see SetMaxTokens):

  • default DefaultReasoningMaxTokens = 2048
  • 0 uses the default
  • a negative value restores the previous uncapped behaviour

A finish_reason=length on a capped step is final, with no length retry:

  • the tool pick continues without the reasoning;
  • the parameter step takes its existing fallback;
  • a complete call that is cut at the cap, and reasoning that finishes normally, are used as before.

All other decisions are unchanged, and the option is forwarded to plan sub-executions.

Tests:

  • Ginkgo specs cover endless tool-pick reasoning, endless parameter reasoning, a short-reasoning counter-check and the option semantics.
  • Internal tests cover the non-streaming path and the unchanged length retry without a cap.
  • Without the fix, 3 of the specs and the non-streaming cap test fail. make test passes (88/88).

🤖 Generated with Claude Code

https://claude.ai/code/session_01Y8ngj32vPUcLs3QKgTYCxy

The reasoning steps of WithForceReasoning (before the tool pick and before
parameter generation) send no output cap of their own, so a model that never
closes the reasoning argument runs to the client's global cap (16384 by
default), and the length retry then doubles it. Observed with gemma4-26b-a4b
behind LocalAI: one forced reasoning call streamed 12,969 tool-call deltas for
478-501 s until the cap, and the turn never answered.

The reasoning steps now carry max_tokens from WithReasoningMaxTokens (default
DefaultReasoningMaxTokens = 2048; 0 uses the default, a negative value keeps
the previous uncapped behaviour). The cap goes on the request, where the
bundled clients take it over their own default (see SetMaxTokens). A
finish_reason=length on such a capped step is final: no length retry, the call
returns an error wrapping errOutputCapReached; a complete tool call that is
cut at the cap is still used. The parameter-reasoning step already falls back
on any error; the tool-pick step now continues without the reasoning on this
error only, other errors stay fatal. A reasoning that finishes normally is
used as before. The option is forwarded to plan sub-executions.

Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de>
Assisted-by: Claude Opus 5.5 <noreply@anthropic.com>
walcz-de added a commit to walcz-de/LocalAI that referenced this pull request Oct 2, 2026
cogito 0400c22: forced reasoning steps get their own output cap (WithReasoningMaxTokens, default 2048) — mudler/cogito#84. A reasoning call ran 478-501 s to the 16384-token cap and the agent never answered.

Assisted-by: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant