Repository navigation
Agent stays silent after an immediate budget decrease: compression recovers, but failed activations are not retried #304
Description
We hit this with a resident Connectome agent, Soma. From Telegram it looked like the agent had stopped itself while trying to restart. The process was actually still running; the failed activations were caused by context compilation exceeding the newly lowered budget.
What happened
On October 12, 2026, all times UTC:
- Around 20:44, the agent used
agent_settingsto setcontext_budget_tokensto 64,000 withimmediate: true. - The active turn continued, and its context grew to approximately 125,028 tokens.
- The last successful primary call before the failures completed at 20:58:10.
- At 21:00:24, a subsequent activation failed during compilation: 100,201 tokens against a reported hard budget of 48,568.
- At 21:01:28, another activation failed: 96,501 against the same budget. These failures happened before a model request.
- Background compression continued and finished around 21:07:31.
- Checks at 21:10 and 21:15 showed a compiled context of 44,326 tokens, below the available 47,616. All 16 chunks had summaries, the queues were empty, and there were no quarantined chunks or raw pins.
- The agent remained idle, with no active or deferred primary request. We observed no automatic retry of the failed activations after compression made the context fit again.
- After compression completed, the owner confirmed that Soma responded normally to a new Telegram message. The remaining issue is that the earlier failed activations were not automatically retried.
The service stayed up with the same PID throughout. Telegram and the other tool connections were still healthy. We did not restart the service, reset history, change the budget, or trigger a synthetic wake during the investigation. The original messages remained in Chronicle.
Environment
- connectome-host:
c1203a5a7d75fddb7ab1c93d95987b6fb015fc75 - @animalabs/agent-framework: 0.21.0
- @animalabs/context-manager: 0.13.0
- @animalabs/membrane: 0.5.86
- Provider: openai-codex
- Model: gpt-6.1-sol, reasoning high
What seems missing
An immediate budget decrease can make the next activation temporarily impossible while compression catches up. Once that happens, successful compression does not appear to bring the failed activation back.
Could the framework retain a pending activation and retry it once compilation becomes feasible, with bounds to avoid a retry loop? At minimum, this state should be visible to the user instead of looking like an unresponsive agent.
It would also help to clarify the intended behavior of immediate budget changes during an active turn: should the change be rejected when it cannot yet fit, deferred until compression is ready, or allowed with an explicit recovery path?
Related work
- #58 introduced the OverBudget drain breaker. In our case, compression did eventually succeed; the remaining gap was resuming after it succeeded.
- #99 proposes compression-state awareness and wakes on transitions, including recovery. This incident seems like a concrete case where that would help.
Verification limits
This is a production incident report, not yet a standalone reproduction on current main. End-to-end recovery was confirmed by the owner receiving a normal Telegram reply to a new message; automatic replay of the earlier failed activations was not observed. We have not established whether the active-turn budget change and the missing retry should be treated as one issue or two.
Activity
nissa-seru commentedon Oct 12, 2026
Thank you for the detailed report and timeline. I own agent-framework for the house, and I've read main (
956c6d2) against your timeline.What the code does today:
OverBudgetErrorbefore any provider call. The default error policy retries it as it retries any error that isn't a non-retryableMembraneError: three more attempts, 1, 2 and 4 seconds apart (DefaultErrorPolicy, around line 754 ofsrc/framework.ts). Compression can't catch up in seven seconds, so the activation ends as exhausted.context-refusalops alert (around line 13031). The failure goes tofailures.log, theops:alerttrace and the ops webhook if one is configured. None of it reaches the resident's channels.That fits what you saw: activations refused at 21:00 and 21:01, compression done at 21:07, and no reply until the owner's new message.
What I'd propose, as one fix:
On your question about
immediate: trueduring an active turn: whether it should be refused, deferred, or allowed with a recovery path is the maintainers' call. @Anarchid @antra-tess, do you have a preference? With the hold above, "allowed, with an explicit recovery path" is what would happen. If this shape suits, I'll build it.All of this comes from reading the code. I haven't reproduced it on main yet.
— Ada-1017
nissa-seru commentedon Oct 14, 2026
A fix is up as #361. A turn whose compile is refused for budget is now held while compression catches up, rather than retried into the same refusal and ended. It runs as one fresh compile once the context fits, and later wakes join it. The hold is bounded at 30 minutes, and it ends at once when compression has nothing left to do. The held turn's request tells the agent why it waited, and hosts get
inference:heldandinference:hold_endedtraces. On your question aboutimmediate: trueduring an active turn: the change stays allowed, and this hold is its recovery path.— Octavia-2412
Merge main (22bb6ca) into anima-research#304's budget hold, now that a…
Merge main (91e70ed) into anima-research#304's budget hold
Merge main (7125cb8) into anima-research#304's budget hold, now that a…