Skip to content

feat(own-loop): in-memory tool-result prune harness safety net - #75

Merged
jonnyparris merged 1 commit into
mainfrom
fix/own-loop-inline-prune
May 25, 2026
Merged

feat(own-loop): in-memory tool-result prune harness safety net#75
jonnyparris merged 1 commit into
mainfrom
fix/own-loop-inline-prune

Conversation

@jonnyparris

Copy link
Copy Markdown
Owner

What

A no-LLM, no-storage harness safety net that keeps the own-loop's in-memory messages array under the context budget when Think-level compaction can't help.

Why

Reproduced this morning by deploying #73 and watching wrangler tail:

  • Gemma 4 26B (and Kimi K2.6) make many tool calls in a single first user turn
  • Each tool result inflates the own-loop's closure messages array
  • Think persists the assistant message only AFTER streamText() finishes, so during the turn realMessages.length === 1
  • Compaction has nothing to summarise; runs as a no-op
  • Hard-stop fires; session ends with 'context budget exhausted'

PR #73 + PR #74 fixed compaction's reachability and silent-no-op bugs. This PR closes the remaining gap by attacking the actual bytes — the own-loop's in-memory tool-result payloads.

How

New module src/own-loop-prune.ts exporting pruneOversizedToolResults(messages, opts):

  • Pure function. No async, no storage.
  • Walks tool messages oldest-first, preserves the most recent N (default 2).
  • Replaces the value of each large tool-result part with a short placeholder string. Keeps the envelope (toolName, toolCallId) intact so the model's prior tool-call still resolves.
  • Stops as soon as estimated tokens drop below target.
  • Configurable: targetTokens, preserveRecentToolMessages, minPrunablePayloadChars.

Wired into coding-agent.ts onChatMessage own-loop, between the pre-step Think compaction attempt and the hard-stop:

projectedTokens >= MID_LOOP_COMPACTION_THRESHOLD
  → maybeCompactContext({ force: true })    // PR #73 + #74
  → if still over: pruneOversizedToolResults  // this PR
  → if still over: hard-stop

Tests

test/own-loop-prune-unit.test.ts — 7 cases:

  • under-budget no-op
  • oldest-largest pruned first
  • preserves the most recent N tool messages regardless of size
  • envelope (toolName, toolCallId) survives pruning
  • skips payloads smaller than minPrunablePayloadChars
  • stops pruning once budget back under target
  • no-op when there are no tool messages

All 21 existing compaction tests still pass; typecheck clean.

Risk

  • Pruned tool results are gone — the model can't re-read them. The placeholder tells the model to re-run the tool if it still needs the data. This trades one ~50k-token tool dump for ~150 chars and the model's discretion to repeat.
  • Affects every session that crosses the mid-loop threshold, not just weak orchestrators. Strong models will rarely hit the path; when they do, the same trade-off applies. The default minPrunablePayloadChars: 2000 keeps small results intact.

Linked

beep-boop-🤖

The harness fix for the failure mode that breaks Gemma + Kimi K2.6 as
orchestrators on first-turn explorations.

## Problem

Think persists the assistant message only AFTER streamText() completes
the whole generator. On a long-running first user turn, sessions.getHistory()
returns just [user] — realMessages.length === 1. The autocompaction
summariser has nothing to summarise.

Meanwhile the own-loop's closure variable `messages` accumulates
tool-result entries across iterations. Weak orchestrators (Gemma 4 26B,
Kimi K2.6) routinely make many tool calls in a single turn whose results
compound the local array to 200k+ tokens. The pre-step hard-stop then
kicks in before the model can finish, ending the session with
"context budget exhausted".

## Fix

New pure helper `pruneOversizedToolResults` in src/own-loop-prune.ts:
- Walks the own-loop's in-memory messages array
- Replaces the largest tool-result payloads with short placeholders
- Preserves the tool-call envelope (toolName, toolCallId) so the
  model's pending tool-call still resolves
- Prunes oldest-first, preserves the most recent N (default 2)
- Stops as soon as projected tokens are back under target

Wired into the pre-step budget check in onChatMessage's own-loop,
running AFTER the Think-level compaction attempt and BEFORE the
hard-stop. The order is intentional: compaction handles previous
turns (cheap when it works), prune handles the current turn (the case
compaction can't fix).

No LLM calls, no storage writes. The placeholder tells the model the
data was pruned and to re-run the tool if it still needs it.

## Why this is the right fix

PR #73 made autocompaction reachable on the first turn.
PR #74 made it actually call into the summariser with force=true.
This PR closes the remaining gap: compaction's input source (Think's
persisted history) is structurally empty on turn 1; the own-loop's
in-memory history is where the bytes actually live.

## Tests

- 7 new unit tests for pruneOversizedToolResults covering: under-budget
  no-op, oldest-largest first, preservation of recent N, envelope
  preservation, min-payload-chars floor, early stop, no tool messages
- All 21 existing compaction tests still pass
- Typecheck clean
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant