Repository navigation
Conversation
A compaction request runs with the context nearly full, and its output
budget is clamped to what is left (down to a 256-token floor). A
thinking model can spend all of it reasoning and return no answer, or a
cut one; the loop then took the reasoning as the summary. Seen live with
a Qwen3-based model behind llama.cpp: two compactions in a row stored the
same 700 characters of reasoning cut mid-sentence ("The user wants me
to summarize the conversation. Let me look at..."); the second window
started from that text and the next compaction copied it.
A summary attempt with no answer, or a cut one, is now retried once in
the model's non-thinking mode (getCurrentModelSettings({ thinking: false
})), after closing the failed attempt and asking for the summary only.
On the retry, an answer is kept even if cut, and a complete
reasoning-only answer is still accepted (setups that route the whole
answer through the reasoning field); otherwise the existing empty-summary
path keeps the full context and reports a recoverable error.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Seen live with llama.cpp (n_ctx 81920): a large tool result pushed the next request to 88,901 tokens. llama.cpp's "exceeds the available context size" error was not recognized as a context overflow, so the very same request was retried with backoff until the turn gave up. - isContextLengthError recognizes llama.cpp's error; promptOverflow() reads when the prompt alone exceeds the window (llama.cpp, or an OpenAI-style error whose messages alone are over), where a smaller maxTokens cannot help. - On such an overflow the loop starts a compaction instead of retrying. - The threshold was only checked on the measured prompt after each response; tool results added since can cross it. It is now also checked before sending, with the estimated tool result tokens. - A compaction request that would leave less than 4,096 tokens for the summary shortens tool results (start and end kept, marker in between) instead of overflowing too. The results before the latest round go first, most recent first: the conversation up to the previous request is in the provider's prompt cache, and shortening near its start made llama.cpp prefill 28-69k tokens again (25-55 s per compaction); near its end only the part after the change is recomputed. The latest round goes last: it is what the agent is about to act on (the compaction often runs right after it arrives), and cutting its middle made the agent re-run the command. After an overflow, the request is sized from what the backend measured (tokens per character of the rejected request): the ~4 characters per token estimate is far off for code (llama.cpp counted ~3), and a request cut on it still overflowed. The measured density is kept for the rest of the turn: later compactions (often started by the pre-send check, with nothing measured) and the pre-send estimate use it too; live, three compactions in a turn each began with an overflow before that. - The overflow retries are a budget per run of failures, reset after a successful request, not per turn: a long turn can compact many times. The existing behavior for an overflow caused by the requested output only (retry with a halved maxTokens) is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
After a sub-agent compaction its context restarted at the summary, which dropped the task it was given (its first user message). Once the sub-agent went on with tool calls the request had no user message at all, and Qwen3-style chat templates reject that: llama.cpp answered HTTP 500 "No user query found in messages" on every retry until the turn was stopped (seen live with a Qwen3-based model). The task is now kept first, before the summary: the summary is the model's paraphrase of the work so far, the task is what was asked. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When a response crossed the compaction threshold, the loop started the compaction at once and dropped that response's tool calls. With a context window not much larger than one tool output this happened at nearly every step: seen live with a 40k window and ~22k-token command outputs, the model's write of the value it had just read was dropped, the summary did not have it, and the agent ran the same commands again and again (1 1 2 1 2 1 2 3 4 5 6 1 1 2 3 for a 1..10 task, 15 compactions). A response with tool calls now runs them first; the pre-send check then compacts before the next request, with their results counted, so the summary includes the work done. A response without tool calls compacts right away as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When a response without tool calls crossed the compaction threshold, the loop appended the compaction prompt without closing that response: its message never got a message.done and stayed "streaming" for good, live and after a reload (seen live: one stuck message per compaction). It is now closed first (segments and interim stats, like the other mid-turn closes) and the client is told it stopped streaming. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The compaction sizing estimated ~4 characters per token and only learned the real density from an overflow error. With a configured context window smaller than the backend's (40k in OpenFox, 81,920 in llama.cpp), nothing ever overflows: dense tool output (logs, code: ~3 characters per token) crossed the threshold unseen and a compaction request reached 64k tokens for a 40k setting. The density is now measured on every response whose request is large enough to be reliable (40k message characters: below, the system prompt and tool definitions skew it), and a measure above the estimate (e.g. only uncached tokens reported) is ignored. The pre-send check and the compaction sizing use it for the rest of the turn. The density counts the whole request: messages, system prompt and tool definitions. Measured on messages only, the fixed part made the smaller summary request come out above its target (live: 37-38k for ~34k). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When the model called a tool while asked for the compaction summary, the loop appended the "tool calls are not possible" reminder without closing that attempt: its message never got a message.done and stayed "streaming" for good, live and after a reload (seen live: one stuck message per such attempt). It is now closed as partial first, like the other discarded attempts. Same when no summary comes at all: the empty-summary path reported the error without closing the last attempt. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A compaction stopped by the user, or cut by a server crash, before its summary leaves its prompt (and the cut attempt) in the context window. The next user message then got a summary for an answer instead of the work: seen live after killing the server mid-compaction, the agent answered "go on" with a summary and stopped, 1 step of 10 done. It only went unnoticed when the context was still over the threshold, since a new compaction then followed. When the context is built, a compaction prompt followed by a message the user wrote is no longer being answered: it is dropped with what follows it up to that message. A prompt still being answered is only followed by system messages (retries, reminders) and is kept. Snapshot replays carry isSystemGenerated and metadata so the same applies to them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When new tool results push the context over the compaction threshold, the summary request added them to the previous request and, when that did not fit, shortened tool results inside it. Either way the request no longer started like the previous one: the backend prefilled most of the conversation again at the worst moment, with the context full. Seen with llama.cpp and Cache Hunter: 28-57k tokens recomputed per compaction (23-51 s), on every compaction of a long tool-heavy turn. The round that crossed the threshold (the assistant message and its tool results, not yet sent) is now left out of the summary request, which is then exactly the previous request plus the compaction prompt. The summary records it (carriedMessageIds) and the next window goes on from the summary and that round, in both the top-level and the sub-agent context (snapshots included). A round too big for that (more than half of what the window leaves after the summary) is summarized with the rest, as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
vv7r
force-pushed
the
fix/compaction-small-context
branch
from
October 10, 2026 21:03
106ee82 to
34685b3
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Compaction fixes found while running long, tool-heavy turns with a local Qwen3-based model on llama.cpp (n_ctx 81,920). On today's
developsuch a turn gets stuck as soon as one tool result pushes a request past the backend's window: the overflow error is not recognized and the same request is retried with backoff forever. Nine commits, one problem each, each with its own test written first (failing before the fix).This is also the sub-agent compaction fix mossaab#2 (follow-up to #401) depends on.
message.doneand stayed "streaming" for good, live and after a reloadLive results
Same model and machine, provider configured with a 80,000-token context and a 0.93 compaction threshold, one session at a time. Each task asks for N commands whose output (14 or 43 KB of logs) hides one secret line, then for the N secrets in order.
developNo errors on this branch; largest prompt 73,896 tokens.
Verification
develop, through the pre-commit hook.AI-Enhanced Development
Cache Impact
develop, request by request. With this branch, summary requests keep the whole previous request as their prefix, and the prefix only breaks where a new context window starts after a summary. When a summary request has to be shortened, the cut is placed as late as possible so most of it stays cached.🤖 Generated with Claude Code