diff --git a/changelog.d/budget-refusal-holds-turn.changed.md b/changelog.d/budget-refusal-holds-turn.changed.md new file mode 100644 index 000000000..e95f91914 --- /dev/null +++ b/changelog.d/budget-refusal-holds-turn.changed.md @@ -0,0 +1,39 @@ +- **Hosts and operators:** a turn held for its context budget (#304, under + Fixed) is a turn alive until it runs or ends, as a turn a provider hold + keeps for its retry is. Before, the refused compile ended the turn within + seconds. + - `getHostModeStatus()` reads `drained: false` while it waits and lists the + agent in `activeTurnAgents`, and wakes that join it count in + `gatedRequests`. A quiesce waits for it: once the hold releases it, the + turn starts and finishes, as any turn alive at the quiesce does, while + the wakes that joined it wait for `resume()`. A quiesce with + `abandon: true` ends it, as do `abortInference()`, `stop()` and a + conversation agent's disposal, each with one `inference:aborted`. After + an abandon or `abortInference()`, the wakes that joined it go back to the + queue and run as a turn of their own. + - A surgery, a branch switch, and an undo or redo that would move the store + are refused (`agent-busy`) while a turn is held, and so is + `puppetToolCall` for the held agent. The refusal names the hold, its + deadline and its reason, and says that a quiesce with `abandon: true` + ends the turn. A host that ran surgery on a resident whose turn had ended + on such a refusal now waits for the hold, up to its 30 minutes, or ends + the turn. + - The turn's terminal event comes when the held turn ends, after a hold of + up to 30 minutes, rather than as an exhaustion within seconds. The + refused attempt emits `inference:failed`, the turn's next start emits + `inference:started` again, and new traces `inference:held` and + `inference:hold_ended` come between. `healthSnapshot()` gives each + agent's `budgetHold` (or null). + - The `context-refusal` ops alert goes out when the hold starts (its + `ops:alert` trace each time, its webhook post at most once per 15 minutes + for an agent), and says the turn is held until its deadline. A hold's + drain kicks log as `[budget-hold]`. +- **Agents:** the turn's next start once its hold ends, with a fit or + without, stores a note in the agent's history saying why it waited, where + the turn reads it, as an `afterUser` context injection is stored (while the + window is held, the compile places it instead): "[budget-hold] Your context + didn't fit its budget (100,201 tokens against 48,568), so this turn waited + 7 min for compression to make room." The note lands before the compile the + preview judged fit, so at the margin it can tip that compile into a + refusal, and the hold then holds the turn again until something it watches + changes. diff --git a/changelog.d/budget-refusal-holds-turn.fixed.md b/changelog.d/budget-refusal-holds-turn.fixed.md new file mode 100644 index 000000000..ab6cc146f --- /dev/null +++ b/changelog.d/budget-refusal-holds-turn.fixed.md @@ -0,0 +1,45 @@ +- A turn whose compile is refused for budget waits for compression and then + runs, instead of ending (#304). The refusal comes before any provider call, + for example after an immediate budget decrease. Before, the error policy + retried it into the same refusal within seconds and ended the turn, and once + compression had made the context fit, nothing ran it again, so the agent + stayed silent until its next wake. Now a resident's or a conversation + agent's turn is held while its context manager has compression pending, and + later wakes join it. The hold leaves the provider gate open, so compression + keeps running. An ephemeral run, a turn in a framework that `start()` + hasn't started (only `runUntilIdle()` runs its turns, so nothing would run + a released one), and a refusal with nothing pending fail as before. What a + held turn changes for hosts is under Changed. + - **When it runs:** as one fresh compile, once context-manager's preview of + that compile, at its budget and with its thinking binding, says it fits. + A dry compile can't say so, since it can resolve a plan the turn's + compile refuses. A refusal the preview rethrows, such as a context + refusal, counts as a refusal. A strategy with no preview gets the dry + compile. The hierarchical strategy's preview reports a context that + doesn't fit, and for one that does, the dry compile decides. The hold + looks after every maintenance pass and every drain kick it starts. It + takes a fresh verdict only once something that could make the context + fit has changed: compression's progress or whether it still has + compression pending, a message added, edited or removed (other than the + arrival of a stored context injection, the hold's own note among them, + or of a clock row), the agent's runtime settings, or the budget its next + compile plans at. A turn released and refused again is held again with + its first deadline, and isn't released again until one of those + changes. + - **When it ends without a fit:** 30 minutes after the refusal that started + it, or at once when compression has nothing pending. The turn then runs + if its context fits by then, and otherwise its refusal takes the error + policy's path, as a refused compile did, its reason saying when it was + first held. The 30 minutes also bound a compression that is pending but + doesn't move. The hold lives in memory: a restart drops it, and the + agent's next wake compiles afresh. + - **Undo:** a hold adds no undo step. When the refused start recorded a + checkpoint, as a turn's first start does, the start the hold releases + moves it to after the messages that arrived while the turn waited, so + the undo keeps them, as it keeps the context injections the refused + start stored, and a hold that's stopped leaves it where it was. + When a retry is refused, an error policy's or a tool-result guard's, the + turn keeps the checkpoint its first start recorded, before what it had + done, and the undo takes the messages that arrived while it waited too. + The redo its new work invalidated stays invalidated, as at any turn's + first start. diff --git a/src/agent.ts b/src/agent.ts index 08b1304a1..2593e9b6d 100644 --- a/src/agent.ts +++ b/src/agent.ts @@ -32,6 +32,9 @@ import { DEFAULT_TOOL_RESULT_INLINE_MAX_CHARS, } from './tool-result-history.js'; +/** ContextManager.compile's own budget when it is given none. */ +const CONTEXT_MANAGER_DEFAULT_BUDGET: TokenBudget = { maxTokens: 100_000, reserveForResponse: 4_000 }; + export interface ActivationRequestOptions { /** The silent heartbeat tick this request answers, if any. */ silentHeartbeat?: SilentHeartbeatTick; @@ -545,11 +548,17 @@ export class Agent { const resolved = this.resolveBudget(budget); const reserve = this.toolResultGuard.pendingWireReserveTokens; if (reserve === 0) return resolved; - // Mirrors ContextManager.compile's own default when no budget is set. - const base = resolved ?? { maxTokens: 100_000, reserveForResponse: 4_000 }; + const base = resolved ?? CONTEXT_MANAGER_DEFAULT_BUDGET; return { ...base, reserveForResponse: base.reserveForResponse + reserve }; } + /** The budget the agent's next compile plans at (compileBudget), for a + * preview of that compile: context-manager's own default when the agent + * sets none, as its compile then gets. */ + nextCompileBudget(): TokenBudget { + return this.compileBudget(undefined) ?? CONTEXT_MANAGER_DEFAULT_BUDGET; + } + /** Structural compatibility keeps lightweight test/host ContextManager * doubles working while making the new capability optional at runtime. */ private getHotContextSettings(): HotContextSettingsStatus | null { diff --git a/src/framework.ts b/src/framework.ts index 62d5b2f6c..e722eb136 100644 --- a/src/framework.ts +++ b/src/framework.ts @@ -1737,6 +1737,11 @@ interface TurnCheckpoint { timestamp: number; } +interface RecordedTurnCheckpoint { + turnIndex: number; + evicted: TurnCheckpoint[]; +} + type DiscordAwarenessDrainOutcome = | { status: 'delivered'; delivered: number; failed: number } | { status: 'unavailable'; accounted: number }; @@ -2091,18 +2096,18 @@ export interface HostModeStatus { * settled teardown, strictly wider than activeStreams), no continuation * of a turn on its way to its start (a budget restart's or a tool-result * guard's retry, queued, held by a provider hold, parked or in a scheduler - * pass), no turn whose call failed into a provider hold waiting for the - * hold to release its retry (wherever that retry then waits), and no wake - * parked on provider admission. It covers the - * framework's own turns: a caller of the direct Agent API - * (`Agent.runInference`) owns its turns' timing. */ + * pass), no turn whose call failed into a provider hold, or whose compile + * was refused for budget, waiting for the hold to release its retry + * (wherever that retry then waits), and no wake parked on provider + * admission. It covers the framework's own turns: a caller of the direct + * Agent API (`Agent.runInference`) owns its turns' timing. */ drained: boolean; activeTurns: number; /** The agents with a turn alive, the ones `activeTurns` counts, sorted: * each from its turn's dequeue to its settled teardown, so a turn in its * hooks or compile, one finishing its deliveries, one whose continuation - * is on its way to its start, or one a provider hold keeps for its retry, - * is listed. These are the agents an abandon or + * is on its way to its start, or one a provider hold or a budget hold + * keeps for its retry, is listed. These are the agents an abandon or * a stop() made now would find mid-turn. */ activeTurnAgents: string[]; /** Wakes that own provider admission while waiting for an in-flight @@ -2114,9 +2119,9 @@ export interface HostModeStatus { /** Inference requests waiting to start: queued (parked by the quiesce wake * gate while quiesced, or, while serving, until the scheduler's next pass * takes them), in a scheduler pass's hands, or held by a provider hold (an - * acceleration or host hold, or a provider's stated wait). There, a start - * that carries a turn on counts as the wakes merged into it. A start parked - * on provider admission behind an auxiliary call counts once in + * acceleration or host hold, or a provider's stated wait) or a budget hold. + * There, a start that carries a turn on counts as the wakes merged into it. + * A start parked on provider admission behind an auxiliary call counts once in * `parkedAdmissions` instead, however many wakes are merged into it. * Continuations of a turn (budget restart, tool results) and a held turn's * retry, which a quiesce lets through, are never counted. */ @@ -2419,6 +2424,28 @@ const TURN_RETRY_SETTINGS: RetrySettings = Object.fromEntries( statedWaitCeilingMs: s.statedWaitCeilingMs === null ? null : Math.max(s.statedWaitCeilingMs, PROVIDER_ACCELERATION_MAX_COOLDOWN_MS), }])) as RetrySettings; const PROVIDER_ACCELERATION_JITTER_MS = 5_000; +const BUDGET_HOLD_MS = 30 * 60_000; +const BUDGET_HOLD_NOTE_NAMESPACE = 'framework:budget-hold'; +type BudgetHoldOutcome = 'fit' | 'check_failed' | 'deadline' | 'nothing_pending' | 'stopped'; +function describeWait(ms: number): string { + return ms < 60_000 ? `${Math.max(1, Math.round(ms / 1000))} s` : `${Math.round(ms / 60_000)} min`; +} +interface BudgetHold { + state: 'holding' | 'released' | 'spent'; + startedAt: number; + until: number; + heldRequests: InferenceRequest[]; + errorType: 'over_budget' | 'context_refusal'; + reason: string; + budget: number | undefined; + actual: number | undefined; + timer?: ReturnType; + recheck?: ReturnType; + checking: boolean; + lookedAt?: string; + ended?: { outcome: 'deadline' | 'nothing_pending' }; + checkpoint: RecordedTurnCheckpoint | undefined; +} /** Longest delay one setTimeout can represent (2^31-1 ms, about 24.8 days); * a longer timer fires almost at once, so longer waits re-arm in steps. */ const MAX_TIMER_DELAY_MS = 2_147_483_647; @@ -3364,6 +3391,11 @@ export class AgentFramework { private providerAccelerationCooldowns: Map = new Map(); private providerAccelerationRecoveries: Map = new Map(); private providerAccelerationLastRecovery: Map = new Map(); + private budgetHolds: Map = new Map(); + private budgetHoldMs = BUDGET_HOLD_MS; + private overBudgetDrainAfter: Map void>> = new Map(); + private budgetHoldRecheckMs = DEFAULT_MAINTENANCE_INTERVAL_MS; + private readonly storeChanges = new WeakMap(); private providerAccelerationDefaultCooldownMs = PROVIDER_ACCELERATION_DEFAULT_COOLDOWN_MS; private providerAccelerationJitterMs = PROVIDER_ACCELERATION_JITTER_MS; private providerHoldHook: ProviderHoldHook | undefined; @@ -5053,6 +5085,7 @@ export class AgentFramework { }); // A stopped host must never hang behind its own cooldown. step(() => this.cancelProviderAdmission()); + for (const name of [...(this.budgetHolds?.keys() ?? [])]) step(() => this.dropBudgetHold(name)); // An open logical turn doesn't survive a stop: a start after a restart // on this instance opens a new one rather than continuing the old turn's // routing (openLogical). A resume still continues its turn, with its @@ -6148,6 +6181,351 @@ export class AgentFramework { statedWaitMs: stated, }; } + private holdForBudgetRefusal( + agent: Agent, + error: Error, + trigger: InferenceRequest | undefined, + checkpoint: RecordedTurnCheckpoint | undefined, + endStart: () => void, + ): boolean { + const { errorType } = this.classifyInferenceError(error); + if (errorType !== 'over_budget' && errorType !== 'context_refusal') return false; + if (!this.budgetHolds || !this.running) return false; + if (this.ephemeralRuns.has(agent.name) || this.agents.get(agent.name) !== agent) return false; + const existing = this.budgetHolds.get(agent.name); + if (existing?.state === 'spent') return false; + const own = asTurnRetry(trigger + ?? { agentName: agent.name, reason: 'budget-hold-release', source: 'framework', timestamp: Date.now() }); + if (existing?.state === 'holding') { + this.holdRequest(existing.heldRequests, own); + if (checkpoint !== undefined) this.dropTurnCheckpoint(agent.name, checkpoint); + endStart(); + return true; + } + if (!this.compressionPending(agent)) return false; + const now = Date.now(); + const figures = error as Error & { budget?: unknown; actual?: unknown }; + const hold: BudgetHold = existing ?? { + state: 'holding', startedAt: now, until: now + this.budgetHoldMs, heldRequests: [], + errorType, reason: '', budget: undefined, actual: undefined, checking: false, checkpoint: undefined, + }; + hold.state = 'holding'; + hold.checkpoint = checkpoint; + hold.heldRequests = [own]; + hold.errorType = errorType; + hold.reason = boundFailureText(String(error.message), FAILURE_SUMMARY_MAX_CHARS); + hold.budget = typeof figures.budget === 'number' ? figures.budget : undefined; + hold.actual = typeof figures.actual === 'number' ? figures.actual : undefined; + hold.timer = setTimeout(() => this.releaseBudgetHold(agent.name, 'deadline'), Math.min(Math.max(0, hold.until - now), MAX_TIMER_DELAY_MS)); + hold.timer.unref?.(); + if (this.maintenanceIntervalMs <= 0) { + hold.recheck = setInterval(() => this.compressForBudgetHold(agent), this.budgetHoldRecheckMs); + hold.recheck.unref?.(); + } + this.budgetHolds.set(agent.name, hold); + endStart(); + this.emitTrace({ + type: 'inference:held', + agentName: agent.name, + errorType, + error: hold.reason, + ...(hold.budget !== undefined ? { budget: hold.budget } : {}), + ...(hold.actual !== undefined ? { actual: hold.actual } : {}), + since: hold.startedAt, + until: hold.until, + heldRequests: hold.heldRequests.length, + }); + if (this.budgetHolds.get(agent.name) !== hold || hold.state !== 'holding') return true; + const deadline = new Date(hold.until).toISOString(); + console.error(`[budget-hold] agent=${agent.name} compile refused (${errorType}): ${hold.reason} \u2014 ` + + `turn held until ${deadline} while compression catches up`); + this.opsAlert( + 'context-refusal', + agent.name, + `compile refused (${errorType}): ${hold.reason} \u2014 the turn is held until ${deadline} while compression ` + + 'catches up, and runs once the context fits; if it never does, raise contextBudgetTokens or overBudgetGraceRatio.', + { skipLog: true }, + ); + const kick = setTimeout(() => this.compressForBudgetHold(agent), 0); + kick.unref?.(); + return true; + } + + private compressionPending(agent: Agent): boolean { + try { + return !agent.getContextManager().isReady(); + } catch { + return false; + } + } + + private checkBudgetHolds(): void { + if (!this.budgetHolds?.size) return; + for (const name of [...this.budgetHolds.keys()]) void this.checkBudgetHold(name); + } + + private async checkBudgetHold(agentName: string): Promise { + const hold = this.budgetHolds?.get(agentName); + const agent = this.agents.get(agentName); + if (hold?.state !== 'holding' || !agent) return; + const pending = this.compressionPending(agent); + const signature = this.budgetHoldSignature(agent, pending); + if (signature !== undefined && signature === hold.lookedAt) return; + let verdict = this.budgetHoldPreview(agent); + if (verdict === undefined) { + if (hold.checking) return; + hold.checking = true; + try { + verdict = await this.budgetHoldDryRun(agent); + } finally { + hold.checking = false; + } + if (this.budgetHolds.get(agentName) !== hold || hold.state !== 'holding') return; + if (this.agents.get(agentName) !== agent) return; + if (verdict === 'refused' && this.budgetHoldSignature(agent) !== signature) return; + } + if (verdict === 'refused' && pending) { + hold.lookedAt = signature; + } else if (verdict === 'refused') { + this.releaseBudgetHold(agentName, 'nothing_pending'); + } else { + hold.lookedAt = signature; + this.releaseBudgetHold(agentName, verdict); + } + } + + private budgetHoldPreview(agent: Agent): 'fit' | 'refused' | 'check_failed' | undefined { + const cm = agent.getContextManager() as unknown as { + previewContext?: (budget: TokenBudget, overrides?: undefined, opts?: { thinkingBinding?: ThinkingBinding }) => { fits?: unknown } | null; + }; + if (typeof cm.previewContext !== 'function') return undefined; + const thinkingBinding = agent.thinkingBindingOption(); + // context-manager's preview answers an over-budget plan with fits: false + // and rethrows any other error. A dry compile can resolve a plan the + // committing compile refuses (SelectOptions.dryRun), so it decides only + // where there is no preview. + try { + const preview = cm.previewContext(agent.nextCompileBudget(), undefined, thinkingBinding ? { thinkingBinding } : undefined); + if (!preview) return undefined; + return preview.fits === true ? 'fit' : 'refused'; + } catch (error) { + // The hierarchical strategy's preview has no fold plan for a context + // that fits: context-manager 0.23.0 throws, 0.23.1 returns null. Its dry + // compile refuses as its compile does. + if (error instanceof Error && error.message.includes('requires adaptiveResolution')) return undefined; + return this.budgetHoldThrownVerdict(error); + } + } + + private async budgetHoldDryRun(agent: Agent): Promise<'fit' | 'refused' | 'check_failed'> { + try { + await this.previewActivation(agent.name); + return 'fit'; + } catch (error) { + return this.budgetHoldThrownVerdict(error); + } + } + + private budgetHoldThrownVerdict(error: unknown): 'refused' | 'check_failed' { + const { errorType } = this.classifyInferenceError(error instanceof Error ? error : new Error(String(error))); + return errorType === 'over_budget' || errorType === 'context_refusal' ? 'refused' : 'check_failed'; + } + + private budgetHoldSignature(agent: Agent, pending = this.compressionPending(agent)): string | undefined { + try { + const cm = agent.getContextManager(); + return JSON.stringify([ + this.contextProgress(cm) ?? null, + // A strategy's progress snapshot needn't show what its readiness reads, + // and some strategies report none. + pending, + this.storeChangeCount(cm), + agent.getRuntimeSettings(), + agent.nextCompileBudget(), + ]); + } catch { + return undefined; + } + } + + private storeChangeCount(cm: ContextManager): number { + let changes = this.storeChanges.get(cm); + if (!changes) { + const counted = { count: 0 }; + cm.onMessage((event) => { + if (event.type === 'add' + && (storedInjectionMarkOf(event.message.metadata) !== undefined || clockStampOf(event.message.metadata) !== undefined)) return; + counted.count++; + }); + this.storeChanges.set(cm, counted); + changes = counted; + } + return changes.count; + } + + private compressForBudgetHold(agent: Agent): void { + if (this.budgetHolds?.get(agent.name)?.state !== 'holding' || this.agents.get(agent.name) !== agent) return; + if (this.providerGateBlocked(agent.name)) { + void this.checkBudgetHold(agent.name); + return; + } + this.kickOverBudgetDrain(agent.name, agent, () => void this.checkBudgetHold(agent.name), 'hold'); + } + + private releaseBudgetHold(agentName: string, outcome: Exclude): void { + const hold = this.budgetHolds?.get(agentName); + if (hold?.state !== 'holding') return; + this.clearBudgetHoldTimers(hold); + const waitedMs = Date.now() - hold.startedAt; + const released = hold.heldRequests; + if (outcome === 'fit' || outcome === 'check_failed') { + hold.state = 'released'; + } else { + hold.state = 'spent'; + hold.ended = { outcome }; + } + hold.heldRequests = []; + this.pendingRequests.push(...released); + console.error(`[budget-hold] agent=${agentName} hold ended (${outcome}) after ${waitedMs}ms; ` + + `${released.length} held request(s) queued for one fresh compile`); + this.emitTrace({ + type: 'inference:hold_ended', + agentName, + outcome, + waitedMs, + heldRequests: released.length, + }); + } + + private budgetHoldNote(hold: BudgetHold): string { + const size = hold.budget !== undefined && hold.actual !== undefined + ? ` (${hold.actual.toLocaleString('en-US')} tokens against ${hold.budget.toLocaleString('en-US')})` + : ''; + return `[budget-hold] Your context didn't fit its budget${size}, so this turn waited ` + + `${describeWait(Date.now() - hold.startedAt)} for compression to make room.`; + } + + private dropBudgetHold(agentName: string): void { + this.takeBudgetHold(agentName)?.report('dropped with it'); + } + + private takeBudgetHold(agentName: string): { requests: InferenceRequest[]; report: (wakes: string) => void } | undefined { + const hold = this.budgetHolds?.get(agentName); + if (!hold) return undefined; + this.budgetHolds.delete(agentName); + if (hold.state !== 'holding') return undefined; + this.clearBudgetHoldTimers(hold); + const waitedMs = Date.now() - hold.startedAt; + const requests = hold.heldRequests; + return { + requests, + report: (wakes) => { + const others = requests.filter((r) => !continuesTurn(r)).length; + console.error(`[budget-hold] agent=${agentName} hold ended (stopped) after ${waitedMs}ms with its turn; ` + + `${others} other held request(s) ${wakes}`); + this.emitTrace({ + type: 'inference:hold_ended', + agentName, + outcome: 'stopped', + waitedMs, + heldRequests: requests.length, + }); + }, + }; + } + + private settleBudgetHold(agentName: string): void { + if (this.budgetHolds?.get(agentName)?.state !== 'holding') this.budgetHolds?.delete(agentName); + } + + private clearBudgetHoldTimers(hold: BudgetHold): void { + clearTimeout(hold.timer); + clearInterval(hold.recheck); + hold.timer = undefined; + hold.recheck = undefined; + } + + private afterBudgetHold(agentName: string, error: Error): Error { + const { errorType } = this.classifyInferenceError(error); + if (errorType !== 'over_budget' && errorType !== 'context_refusal') return error; + const hold = this.budgetHolds?.get(agentName); + if (hold?.state === 'released') { + const agent = this.agents.get(agentName); + const outcome = hold.until <= Date.now() ? 'deadline' + : agent && !this.compressionPending(agent) ? 'nothing_pending' : undefined; + if (outcome) { + hold.state = 'spent'; + hold.ended = { outcome }; + } + } + if (hold?.state !== 'spent' || !hold.ended) return error; + const why = hold.ended.outcome === 'deadline' + ? `the hold's ${describeWait(this.budgetHoldMs)} bound passed` + : 'compression had nothing left to do'; + error.message = `${error.message} (the turn was first held ${describeWait(Date.now() - hold.startedAt)} ago; ${why} and it still didn't fit)`; + return error; + } + + private kickOverBudgetDrain( + agentName: string, + agent: Agent, + after?: () => void, + by: 'breaker' | 'hold' = 'breaker', + ): 'not kicked: the framework is stopping' | 'not kicked: a surgery holds the store' | 'already draining' | 'kicking' { + // Like a maintenance pass: none starts once stop() has begun, it starts + // no tick while a surgery holds the store or once its agent has been + // disposed, and a surgery, a switchBranch or stop() waits for the tick + // in flight. It joins the drains they wait for before the breaker's alert is + // raised and before its body runs, so the alert's listeners find it: + // one that begins stop() or a surgery waits for it, and its body then + // ticks none and says why. So does a stop() made from inside its first + // tick, before that tick's first await. + if (this.providerAdmissionClosed) return 'not kicked: the framework is stopping'; + if (this.surgeryHold) return 'not kicked: a surgery holds the store'; + if (after) { + const waiting = this.overBudgetDrainAfter.get(agentName); + if (waiting) waiting.push(after); + else this.overBudgetDrainAfter.set(agentName, [after]); + } + if (this.overBudgetDrainInFlight.has(agentName)) return 'already draining'; + this.overBudgetDrainInFlight.add(agentName); + const tag = by === 'hold' ? '[budget-hold]' : '[inference-failed]'; + const label = by === 'hold' ? 'budget hold' : 'OverBudget breaker'; + const drain = Promise.resolve().then(async () => { + let ticks = 0; + try { + const cm = agent.getContextManager(); + while (ticks < 8 && !this.providerAdmissionClosed && !this.surgeryHold && this.agents.get(agentName) === agent) { + await cm.tick(); + ticks++; + } + const stopped = ticks >= 8 ? '' + : this.agents.get(agentName) !== agent ? ', stopped as the agent was disposed' + : this.providerAdmissionClosed ? ', stopped as the framework stops' + : ', stopped while a surgery holds the store'; + console.error(`${tag} drain kicked for ${agentName} (${label}, ${ticks} ticks${stopped})`); + } catch (err) { + if (err instanceof MaintenanceWithdrawnError) { + console.error(`${tag} drain kicked for ${agentName} (${label}, ${ticks} ticks, stopped: ${err.message})`); + } else if (this.syncMoveReinit && err instanceof Error && refusedForMovedBranch(err.message)) { + console.error(`${tag} drain kicked for ${agentName} (${label}, ${ticks} ticks, stopped: ${this.syncMoveReinit.after === 'undo' ? 'an undo' : 'a redo'} moved the store under the tick: ${err.message})`); + } else { + console.error(`${tag} drain kick failed for ${agentName} after ${ticks} ticks: ${describeFailureWithFrames(err)}`); + } + } finally { + this.overBudgetDrainInFlight.delete(agentName); + } + const waiting = this.overBudgetDrainAfter?.get(agentName); + if (waiting) { + this.overBudgetDrainAfter.delete(agentName); + for (const run of waiting) run(); + } + }); + this.overBudgetDrains.add(drain); + void drain.finally(() => { this.overBudgetDrains.delete(drain); }); + return 'kicking'; + } + /** When a hold's timer next fires: at its deadline, stepped past what one * timer can hold. A hold with no deadline (a wait held until released, or * the stand-in for unreadable records) asks again as often as unreadable @@ -6524,6 +6902,7 @@ export class AgentFramework { this.maintenancePass = pass; void pass.finally(() => { if (this.maintenancePass === pass) this.maintenancePass = null; + this.checkBudgetHolds(); }); begin(this.runQueuedMaintenance().catch((error) => { // The alert cuts its own copies from the whole message, so the @@ -8218,6 +8597,7 @@ export class AgentFramework { // operator tests build one, sets none of these. for (const r of this.pendingRequests ?? []) add(r); for (const cooldown of this.providerAccelerationCooldowns?.values() ?? []) for (const r of cooldown.heldRequests) add(r); + for (const hold of this.budgetHolds?.values() ?? []) for (const r of hold.heldRequests) add(r); for (const r of this.parkedContinuations ?? []) add(r); for (const pass of this.passRequests ?? []) for (const requests of pass.values()) for (const r of requests) add(r); return agents; @@ -8235,14 +8615,22 @@ export class AgentFramework { return this.activeTurnTokens.has(agentName) || this.pendingContinuationAgents().has(agentName); } - /** The provider hold keeping `agentName`'s turn for its retry (TURN_RETRY), - * in words naming its end and its reason, and whether a stated wait holds - * it; undefined when no hold keeps one. */ + /** The provider or budget hold keeping `agentName`'s turn for its retry + * (TURN_RETRY), in words naming its end and its reason, and whether a + * stated wait holds it; undefined when no hold keeps one. */ private heldTurnRetry(agentName: string): { says: string; statedWait: boolean } | undefined { // Optional, as pendingContinuationAgents reads it: a framework built from // its prototype, as some suites build one, has no map of holds. const hold = this.providerAccelerationCooldowns?.get(agentName); - if (!hold?.heldRequests.some(isTurnRetry)) return undefined; + if (!hold?.heldRequests.some(isTurnRetry)) { + const budget = this.budgetHolds?.get(agentName); + if (!budget?.heldRequests.some(isTurnRetry)) return undefined; + return { + says: `a budget hold keeps its turn to start again once its context fits, or at ` + + `${new Date(budget.until).toISOString()} at the latest: ${budget.reason}`, + statedWait: false, + }; + } // A deadline past what a Date can name (an infinite wait, or a stated // one too long for an instant) is one only a release ends. const end = new Date(hold.until); @@ -8270,6 +8658,7 @@ export class AgentFramework { ...this.pendingRequests, ...[...this.passRequests].flatMap((pass) => [...pass.values()].flat()), ...[...this.providerAccelerationCooldowns.values()].flatMap((c) => c.heldRequests), + ...[...(this.budgetHolds?.values() ?? [])].flatMap((h) => h.heldRequests), ].flatMap((r) => mergedBatch(r) ?? [r]).filter((r) => !passesQuiesce(r)).length, backgroundScripts: this.codeExecution?.scripts.leftRunning() ?? 0, deferredWrites: this.deferredMessages.length, @@ -12539,8 +12928,8 @@ export class AgentFramework { for (const [name, a] of this.agents) { if (a.state.status !== 'idle') { busy.push(`${name} is ${a.state.status}`); continue; } if (!this.hasTurn(name)) continue; - // A provider hold keeping the turn's retry is named: a quiesce alone - // doesn't end that turn. + // A provider or budget hold keeping the turn's retry is named: a + // quiesce alone doesn't end that turn. const held = this.heldTurnRetry(name); if (held) { heldTurns = true; @@ -13524,7 +13913,7 @@ export class AgentFramework { // Turn checkpoint internals // ========================================================================== - private recordTurnCheckpoint(agentName: string): void { + private recordTurnCheckpoint(agentName: string): RecordedTurnCheckpoint { const turnIndex = this.turnCounters.get(agentName) ?? 0; this.turnCounters.set(agentName, turnIndex + 1); @@ -13540,11 +13929,21 @@ export class AgentFramework { checkpoints.push(checkpoint); // Trim to max depth - if (checkpoints.length > MAX_TURN_CHECKPOINTS) { - checkpoints.splice(0, checkpoints.length - MAX_TURN_CHECKPOINTS); - } + const evicted = checkpoints.length > MAX_TURN_CHECKPOINTS + ? checkpoints.splice(0, checkpoints.length - MAX_TURN_CHECKPOINTS) + : []; this.saveTurnCheckpoints(agentName, checkpoints); + return { turnIndex, evicted }; + } + + private dropTurnCheckpoint(agentName: string, recorded: RecordedTurnCheckpoint): void { + const checkpoints = this.getTurnCheckpoints(agentName); + if (checkpoints.at(-1)?.turnIndex !== recorded.turnIndex) return; + checkpoints.pop(); + checkpoints.unshift(...recorded.evicted); + this.saveTurnCheckpoints(agentName, checkpoints); + this.turnCounters.set(agentName, recorded.turnIndex); } /** @@ -15428,6 +15827,7 @@ export class AgentFramework { `${cooldown.waitModel !== undefined ? ` on ${cooldown.waitModel}` : ''}: ` + `${cooldown.heldRequests.length} held request(s) dropped with it (the recorded wait stays)`); } + this.dropBudgetHold(agentName); this.emitTrace({ type: 'mcpl:conversation-disposed', agentName, @@ -16398,6 +16798,9 @@ export class AgentFramework { for (const cooldown of this.providerAccelerationCooldowns.values()) { cooldown.heldRequests = cooldown.heldRequests.filter((r) => r.coalescingSubject !== subject); } + for (const hold of this.budgetHolds?.values() ?? []) { + hold.heldRequests = hold.heldRequests.filter((r) => r.coalescingSubject !== subject); + } // A wake of the subject's merged into a start that carries a turn on // (MERGED_BATCH) is withdrawn from the start's batch, so no stop puts // it back and no pass unpacks it. @@ -16836,6 +17239,11 @@ export class AgentFramework { continue; } + const budgetHold = this.budgetHolds?.get(agentName); + if (budgetHold?.state === 'holding') { + for (const req of requests) this.holdRequest(budgetHold.heldRequests, req); + continue; + } const providerCooldown = this.providerAccelerationCooldowns?.get(agentName); if (providerCooldown) { // A provider wait binds its own model; with a different primary model @@ -20444,6 +20852,7 @@ export class AgentFramework { // gate event buffers, for every agent. A newer start holding the token // owns the window now, so it's left alone. if (this.activeTurnTokens.get(agent.name) === turnToken) this.eventGate?.onInferenceEnded(agent.name); + this.settleBudgetHold(agent.name); throw error; } finally { // This start's checks are behind it, whether it handed its turn to a @@ -20475,10 +20884,11 @@ export class AgentFramework { /** * Drop every continuation of `agentName`'s turn still on its way, wherever * it waits: the queue, a provider hold (which, left empty, then releases - * with nothing to run), a park on admission (whose admission it gives - * back), or a scheduler pass's hands (which then never reach it). If one - * was there, the turn ends (endContinuedTurn), unless `drop` is null: the - * caller has ended the turn already. + * with nothing to run), a budget hold (which ends with it), a park on + * admission (whose admission it gives back), or a scheduler pass's hands + * (which then never reach it). If one was there, the turn ends + * (endContinuedTurn), unless `drop` is null: the caller has ended the turn + * already. */ private dropContinuations(agentName: string, drop: ContinuationDrop | null): void { const isContinuation = (r: InferenceRequest): boolean => r.agentName === agentName && continuesTurn(r); @@ -20507,6 +20917,12 @@ export class AgentFramework { hold.heldRequests = take(hold.heldRequests); if (held > 0 && hold.heldRequests.length === 0) hold.emptiedByDrop = true; } + const budgetHold = this.budgetHolds?.get(agentName)?.heldRequests.some(isContinuation) + ? this.takeBudgetHold(agentName) : undefined; + if (budgetHold) { + const wakes = take(budgetHold.requests); + if (putsBack) back.push(...wakes); + } for (const parked of [...this.parkedContinuations]) { if (!isContinuation(parked)) continue; this.parkedContinuations.delete(parked); @@ -20522,13 +20938,18 @@ export class AgentFramework { requests.push(...left); } this.pendingRequests.push(...back); - if (!dropped || !drop) return; + const reportHold = (): void => budgetHold?.report(putsBack ? 'put back in the queue' : 'dropped with it'); + if (!dropped || !drop) return reportHold(); // A primary a provider hold left pending for what it released won't run // now, when nothing else of the agent's waits (endPendingPrimary). - if (!this.pendingRequests.some((r) => r.agentName === agentName) && !this.providerAccelerationCooldowns.get(agentName)?.heldRequests.length) { - this.endPendingPrimary(agentName); + try { + if (!this.pendingRequests.some((r) => r.agentName === agentName) && !this.providerAccelerationCooldowns.get(agentName)?.heldRequests.length) { + this.endPendingPrimary(agentName); + } + this.endContinuedTurn(agentName, drop); + } finally { + reportHold(); } - this.endContinuedTurn(agentName, drop); } /** @@ -20545,6 +20966,7 @@ export class AgentFramework { */ private endContinuedTurn(agentName: string, drop: ContinuationDrop): void { console.error(`[turn-continuation] ${agentName}: its turn ends without its continuation: ${drop.why}`); + this.settleBudgetHold(agentName); const agent = this.agents.get(agentName); const startedAt = this.lastInferenceAt.get(agentName)?.startedAt ?? Date.now(); if (agent && drop.kind !== 'dropped') { @@ -20790,6 +21212,7 @@ export class AgentFramework { } else { this.settleFrameworkCancel({ ...common, kind: stop.kind }); } + this.settleBudgetHold(agent.name); const last = this.lastInferenceAt.get(agent.name); if (last?.startedAt === turnStartedAt) this.lastInferenceAt.set(agent.name, { ...last, endedAt: Date.now() }); } @@ -20985,8 +21408,15 @@ export class AgentFramework { } // Record turn checkpoint before inference (only on first attempt, not retries) + let checkpoint: RecordedTurnCheckpoint | undefined; if (attempt === 0 && trigger?.reason !== 'tool_result_guard_retry') { - this.recordTurnCheckpoint(agent.name); + const hold = this.budgetHolds?.get(agent.name); + if (hold === undefined || hold.state === 'holding') { + checkpoint = this.recordTurnCheckpoint(agent.name); + } else if (hold.checkpoint !== undefined) { + this.dropTurnCheckpoint(agent.name, hold.checkpoint); + checkpoint = this.recordTurnCheckpoint(agent.name); + } this.redoStacks.delete(agent.name); // new work invalidates redo } @@ -21235,6 +21665,17 @@ export class AgentFramework { } } + const heldTurn = this.budgetHolds?.get(agent.name); + const holdNote = heldTurn && heldTurn.state !== 'holding' ? this.budgetHoldNote(heldTurn) : undefined; + if (holdNote !== undefined) { + const noteInjection: ContextInjection = { + namespace: BUDGET_HOLD_NOTE_NAMESPACE, + position: 'afterUser', + content: [{ type: 'text', text: holdNote }], + }; + injections = injections ? [...injections, noteInjection] : [noteInjection]; + } + // An ephemeral watchdog may dispose this Agent while hooks/compile await, // and stop() may close provider admission. Do not create a provider // stream after its generation lost ownership, or once admission closed, @@ -21393,6 +21834,7 @@ export class AgentFramework { else this.turnPrompts.delete(frameTrigger); } + this.settleBudgetHold(agent.name); // The turn has a live stream from here, which carries a stop: this // start's checks are behind it, so its window closes now, before // driveStream can report a stream that fails at once. @@ -21426,7 +21868,7 @@ export class AgentFramework { // a hook/compile/stream-setup failure must not leave "typing…" stuck. // (Retries below restart it; stopTyping is idempotent.) this.channelRegistry?.stopTyping(); - const err = error instanceof Error ? error : new Error(String(error)); + const err = this.afterBudgetHold(agent.name, error instanceof Error ? error : new Error(String(error))); this.emitTrace({ type: 'inference:failed', agentName: agent.name, @@ -21464,6 +21906,10 @@ export class AgentFramework { } const retry = trigger && withTurnCarry(trigger, carry); + if (this.holdForBudgetRefusal(agent, err, retry, checkpoint, + () => this.endHeldStart(agent, checkWindow, turnToken, turnStartedAt, routing))) { + return false; + } if (ownsProviderGate && this.holdProviderAcceleration(agent, err, retry)) { // Capacity scheduling is not poisoned history. The cooldown timer owns // one later fresh compile; no immediate retry or exhaustion marker. @@ -21541,6 +21987,7 @@ export class AgentFramework { speech: '', ...(statedWait === 'fail' ? this.statedWaitNotSlept(err) : { error: err.message }), }, routing); + this.settleBudgetHold(agent.name); if (isRearmed(trigger) && isTransientFailure(err)) this.exhaustingRearms.add(agent.name); this.emitTrace({ type: 'inference:exhausted', @@ -24072,8 +24519,9 @@ export class AgentFramework { } if (agent.state.status !== 'idle' || this.hasTurn(agentName)) { const shown = agent.state.status === 'idle' ? 'idle+turn-alive' : agent.state.status; - // A turn a provider hold keeps for its retry has no stream: the refusal - // names the hold, and how to go on sooner, as surgery's does. + // A turn a provider or budget hold keeps for its retry has no stream: + // the refusal names the hold, and how to go on sooner, as surgery's + // does. const held = agent.state.status === 'idle' ? this.heldTurnRetry(agentName) : undefined; throw new Error(held ? `puppet refused: agent ${agentName} is idle+turn-alive (${held.says}; requires idle); ` + @@ -28282,6 +28730,17 @@ export class AgentFramework { }; })(), lastInference: this.lastInferenceAt.get(name) ?? null, + budgetHold: (() => { + const hold = this.budgetHolds?.get(name); + return hold?.state !== 'holding' ? null : { + since: hold.startedAt, + until: hold.until, + heldRequests: hold.heldRequests.length, + errorType: hold.errorType, + ...(hold.budget !== undefined ? { budget: hold.budget } : {}), + ...(hold.actual !== undefined ? { actual: hold.actual } : {}), + }; + })(), refusalStats: this.refusalStats.get(name) ?? null, })), }; @@ -28712,48 +29171,7 @@ export class AgentFramework { // runUntilIdle() has no maintenance timer, so a drain is its recovery. const drainState = !overBudget ? undefined : !agent ? 'unavailable (no agent handle)' - : this.providerAdmissionClosed ? 'not kicked: the framework is stopping' - : this.surgeryHold ? 'not kicked: a surgery holds the store' - : this.overBudgetDrainInFlight.has(agentName) ? 'already draining' - : 'kicking'; - if (agent && drainState === 'kicking') { - this.overBudgetDrainInFlight.add(agentName); - // Like a maintenance pass: none starts once stop() has begun, it starts - // no tick while a surgery holds the store or once its agent has been - // disposed, and a surgery, a switchBranch or stop() waits for the tick - // in flight. It joins the drains they wait for before the alert is - // raised and before its body runs, so the alert's listeners find it: - // one that begins stop() or a surgery waits for it, and its body then - // ticks none and says why. So does a stop() made from inside its first - // tick, before that tick's first await. - const drain = Promise.resolve().then(async () => { - let ticks = 0; - try { - const cm = agent.getContextManager(); - while (ticks < 8 && !this.providerAdmissionClosed && !this.surgeryHold && this.agents.get(agentName) === agent) { - await cm.tick(); - ticks++; - } - const stopped = ticks >= 8 ? '' - : this.agents.get(agentName) !== agent ? ', stopped as the agent was disposed' - : this.providerAdmissionClosed ? ', stopped as the framework stops' - : ', stopped while a surgery holds the store'; - console.error(`[inference-failed] drain kicked for ${agentName} (OverBudget breaker, ${ticks} ticks${stopped})`); - } catch (err) { - if (err instanceof MaintenanceWithdrawnError) { - console.error(`[inference-failed] drain kicked for ${agentName} (OverBudget breaker, ${ticks} ticks, stopped: ${err.message})`); - } else if (this.syncMoveReinit && err instanceof Error && refusedForMovedBranch(err.message)) { - console.error(`[inference-failed] drain kicked for ${agentName} (OverBudget breaker, ${ticks} ticks, stopped: ${this.syncMoveReinit.after === 'undo' ? 'an undo' : 'a redo'} moved the store under the tick: ${err.message})`); - } else { - console.error(`[inference-failed] drain kick failed for ${agentName} after ${ticks} ticks: ${describeFailureWithFrames(err)}`); - } - } finally { - this.overBudgetDrainInFlight.delete(agentName); - } - }); - this.overBudgetDrains.add(drain); - void drain.finally(() => { this.overBudgetDrains.delete(drain); }); - } + : this.kickOverBudgetDrain(agentName, agent); if (overBudget) { this.opsAlert( 'context-refusal', diff --git a/src/types/trace.ts b/src/types/trace.ts index e3daff6fa..adca9acab 100644 --- a/src/types/trace.ts +++ b/src/types/trace.ts @@ -150,6 +150,57 @@ export type TraceEvent = /** The provider's own error code, when it sent one (at most 128 characters). */ providerErrorCode?: string; }) + | (TraceEventBase & { + /** + * A turn whose compile was refused for budget is held while compression + * makes its context fit, instead of failing (#304). It follows the + * refused attempt's `inference:failed`, as a provider hold does. Its + * logical turn stays open; `inference:hold_ended` follows, and then, + * unless the turn was stopped while held, its fresh compile. After a + * hold that ended without a fit, a refusal of that compile takes the + * error policy's path. + */ + type: 'inference:held'; + agentName: string; + /** 'over_budget', or 'context_refusal' when no summary covers a span the compile would drop. */ + errorType: 'over_budget' | 'context_refusal'; + /** The refusal's message, bounded (300 characters). */ + error: string; + /** The refused compile's budget and size in tokens, when context-manager gave them. */ + budget?: number; + actual?: number; + /** When the hold started (epoch ms): this refusal, or for a released + * turn refused again, the refusal that first held it. */ + since: number; + /** The hold's deadline (epoch ms), 30 minutes after `since`. */ + until: number; + /** Requests the hold has taken so far, the refused one included. */ + heldRequests: number; + }) + | (TraceEventBase & { + type: 'inference:hold_ended'; + agentName: string; + /** + * 'fit': context-manager's preview of the turn's next compile says it + * fits (a dry compile does, for a strategy with no preview), and the + * held requests are queued for one fresh compile. 'check_failed': that + * check failed for a reason other than the budget, and the requests are + * queued for the turn's own compile to meet it. 'deadline': the 30 + * minutes passed. 'nothing_pending': compression has nothing pending + * and the context still doesn't fit. After those two the requests are + * queued too, for one fresh compile, and a refusal of it takes the error + * policy's path, as a refused compile did before the hold. 'stopped': + * the turn ended while it was held (an abandon, a stop of the turn or of + * the framework, or the agent's leaving), reported after its + * `inference:aborted`, once the wakes that joined it are back in the + * queue (after an abandon or a stop of the turn) or gone with it + * (otherwise). + */ + outcome: 'fit' | 'check_failed' | 'deadline' | 'nothing_pending' | 'stopped'; + /** Since the hold started (`inference:held`'s `since`). */ + waitedMs: number; + heldRequests: number; + }) // Streaming inference lifecycle | (TraceEventBase & { diff --git a/test/budget-refusal-hold.test.ts b/test/budget-refusal-hold.test.ts new file mode 100644 index 000000000..ec814105c --- /dev/null +++ b/test/budget-refusal-hold.test.ts @@ -0,0 +1,2356 @@ +import { describe, it } from 'node:test'; +import assert from 'node:assert/strict'; +import { mkdtempSync, rmSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { AutobiographicalStrategy, ContextManager, PassthroughStrategy } from '@animalabs/context-manager'; +import type { EventResponse, InferencePolicy, Module, ProcessEvent, ToolDefinition, ToolResult, TraceEvent } from '../src/index.js'; +import { AgentFramework } from '../src/index.js'; +import type { NormalizedRequest, NormalizedResponse, YieldingStream } from '@animalabs/membrane'; +import { createMockResponse, MockMembrane, MockYieldingStream } from './helpers/mock-membrane.js'; + +class FailsAfterToolsStream extends MockYieldingStream { + override provideToolResults(results: unknown[], options?: Parameters[1]): void { + super.provideToolResults(results, options); + (this as unknown as { events: unknown[] }).events.push({ type: 'error', error: new Error('stream broke') }); + } +} + +class ScriptedStreamMembrane extends MockMembrane { + constructor(private readonly scripts: Array>) { super(); } + override streamYielding(request: NormalizedRequest, _options?: unknown): YieldingStream { + this.calls.push(request); + const script = this.scripts.shift() ?? []; + const rounds = script.filter((r): r is NormalizedResponse => r !== 'fails'); + const stream = script.includes('fails') ? new FailsAfterToolsStream(rounds) : new MockYieldingStream(rounds); + this.lastStream = stream; + return stream; + } +} + +class SequentialStreamMembrane extends MockMembrane { + private streamIdx = 0; + override streamYielding(request: NormalizedRequest, _options?: unknown): YieldingStream { + this.calls.push(request); + const stream = new MockYieldingStream(this.responses.slice(this.streamIdx, ++this.streamIdx)); + this.lastStream = stream; + return stream; + } +} + +async function pollUntil(cond: () => boolean, timeoutMs: number): Promise { + const deadline = Date.now() + timeoutMs; + while (Date.now() < deadline) { + if (cond()) return true; + await new Promise((r) => setTimeout(r, 25)); + } + return cond(); +} + +function stored(framework: AgentFramework): Array<{ text: string; metadata: Record }> { + const cm = framework.getAgent('prime')?.getContextManager(); + const msgs = (cm?.queryMessages({}).messages ?? []) as unknown as Array<{ + content: Array<{ type: string; text?: string }>; + metadata?: Record; + }>; + return msgs.map((m) => ({ + text: (m.content ?? []).filter((b) => b.type === 'text' && typeof b.text === 'string').map((b) => b.text).join('\n'), + metadata: m.metadata ?? {}, + })); +} +const storedText = (framework: AgentFramework): string => stored(framework).map((m) => m.text).join('\n'); +const toolBlocks = (framework: AgentFramework): number => { + const msgs = (framework.getAgent('prime')?.getContextManager().queryMessages({}).messages ?? []) as unknown as Array<{ content?: Array<{ type: string }> }>; + return msgs.flatMap((m) => m.content ?? []).filter((b) => b.type === 'tool_use' || b.type === 'tool_result').length; +}; +const turnOver = (t: { of: (type: string) => unknown[]; internal: { activeTurnTokens: Map } }, completed = 1): Promise => + pollUntil(() => t.of('inference:completed').length >= completed && !t.internal.activeTurnTokens.has('prime'), 5_000); + +async function errorLines(run: (lines: string[]) => Promise): Promise { + const lines: string[] = []; + const original = console.error; + console.error = (...args: unknown[]) => { + lines.push(args.map(String).join(' ')); + original(...args); + }; + try { + return await run(lines); + } finally { + console.error = original; + } +} + +interface Backlog { + ticks: number; fitAt: number; readyAt: number; fit: boolean; + refused: boolean; + alwaysPending: boolean; stalled: boolean; previewless: boolean; refuseCompiles: number; + previews: number; dryRuns: number; dryRunsRunning: number; mostDryRunsRunning: number; compiles: number; + dryRunsAcrossFit: number; + noProgress: boolean; + progress?: number; + refusalName?: string; + /** A dry compile resolves whether the context fits or not, as + * context-manager's resolves an over-budget plan (SelectOptions.dryRun). */ + dryRunResolves?: boolean; + previewError?: Error; dryRunGate?: Promise; dryRunMs?: number; slowCompile?: { nth: number; ms: number }; + compileGate?: { nth: number; until: Promise }; + bindings: { compile: unknown[]; dryRun: unknown[]; preview: unknown[] }; + thrown: Error[]; previewBudgets: unknown[]; +} +const fits = (b: Backlog): boolean => b.fit || b.ticks >= b.fitAt; +const pending = (b: Backlog): boolean => b.alwaysPending || (!fits(b) && b.ticks < b.readyAt); +let refusalName = 'OverBudgetError'; +const overBudget = (actual = 100_201) => Object.assign( + new Error('Compile plan would exceed hard budget: head=41200 tail=8100 middle=62000 budget=48568'), + { name: refusalName, budget: 48_568, actual }, +); + +type Patchable = Record<'compile' | 'tick' | 'isReady' | 'previewContext', (...args: unknown[]) => unknown>; + +async function withBacklog(backlog: Partial, run: (b: Backlog) => Promise): Promise { + const b: Backlog = { + ticks: 0, fitAt: Infinity, readyAt: Infinity, fit: false, refused: false, + alwaysPending: false, stalled: false, previewless: false, refuseCompiles: 0, + previews: 0, dryRuns: 0, dryRunsRunning: 0, mostDryRunsRunning: 0, compiles: 0, + dryRunsAcrossFit: 0, noProgress: false, bindings: { compile: [], dryRun: [], preview: [] }, + thrown: [], previewBudgets: [], + ...backlog, + }; + refusalName = b.refusalName ?? 'OverBudgetError'; + const proto = ContextManager.prototype as unknown as Patchable; + const original = { compile: proto.compile, tick: proto.tick, isReady: proto.isReady, previewContext: proto.previewContext }; + const passthrough = PassthroughStrategy.prototype as unknown as { getProgressSnapshot?: () => unknown }; + proto.compile = async function (this: unknown, ...args: unknown[]) { + if ((args[2] as { dryRun?: boolean } | undefined)?.dryRun === true) { + b.bindings.dryRun.push((args[2] as { thinkingBinding?: unknown }).thinkingBinding); + b.dryRuns++; + b.mostDryRunsRunning = Math.max(b.mostDryRunsRunning, ++b.dryRunsRunning); + try { + const fitsNow = fits(b); + if (b.dryRunGate) await b.dryRunGate; + if (b.dryRunMs) await new Promise((r) => setTimeout(r, b.dryRunMs)); + if (!fitsNow && fits(b)) b.dryRunsAcrossFit++; + if (!fitsNow && !b.dryRunResolves) throw overBudget(); + } finally { + b.dryRunsRunning--; + } + return original.compile.apply(this, args); + } + b.compiles++; + b.bindings.compile.push((args[2] as { thinkingBinding?: unknown } | undefined)?.thinkingBinding); + if (b.slowCompile?.nth === b.compiles) await new Promise((r) => setTimeout(r, b.slowCompile!.ms)); + if (b.compileGate?.nth === b.compiles) await b.compileGate.until; + if (!fits(b)) { + b.refused = true; + b.thrown.push(overBudget()); + throw b.thrown.at(-1); + } + if (b.refuseCompiles > 0) { + b.refuseCompiles--; + b.refused = true; + b.thrown.push(overBudget(49_000)); + throw b.thrown.at(-1); + } + return original.compile.apply(this, args); + }; + proto.previewContext = function (...args: unknown[]) { + b.previews++; + b.previewBudgets.push(args[0]); + b.bindings.preview.push((args[2] as { thinkingBinding?: unknown } | undefined)?.thinkingBinding); + if (b.previewError) { + const error = b.previewError; + b.previewError = undefined; + throw error; + } + if (b.previewless) return null; + // As context-manager's preview does: an over-budget plan answers + // fits: false, and any other refusal, a context refusal among them, is + // rethrown. + if (!fits(b) && refusalName !== 'OverBudgetError') throw overBudget(); + return { fits: fits(b) }; + }; + proto.tick = async function () { if (b.refused) b.ticks++; }; + proto.isReady = function () { return !pending(b); }; + if (!b.noProgress) { + passthrough.getProgressSnapshot = function () { + return { chunksCompressed: b.progress ?? (b.stalled ? 0 : b.ticks), fits: fits(b) }; + }; + } + try { + return await run(b); + } finally { + Object.assign(proto, original); + delete passthrough.getProgressSnapshot; + refusalName = 'OverBudgetError'; + } +} + +class InboxModule implements Module { + readonly name = 'inbox'; + gatherContext?: () => Promise }>>; + constructor(context?: () => string) { + if (context) this.gatherContext = async () => [{ namespace: 'inbox:status', position: 'afterUser', content: [{ type: 'text', text: context() }] }]; + } + async start(): Promise {} + async stop(): Promise {} + getTools() { return []; } + async handleToolCall(): Promise { throw new Error('no tools'); } + async onProcess(event: ProcessEvent): Promise { + if (event.type !== 'external-message') return {}; + return { + addMessages: [{ participant: 'User', content: (event as { content: unknown }).content as never }], + requestInference: true, + }; + } +} + +class BlobToolModule implements Module { + readonly name = 'canned'; + constructor(private readonly duringCall?: () => unknown) {} + async start(): Promise {} + async stop(): Promise {} + getTools(): ToolDefinition[] { + return [{ name: 'fetch', description: 'blob', inputSchema: { type: 'object', properties: {} } }]; + } + async handleToolCall(): Promise { + await this.duringCall?.(); + return { success: true, data: { blob: 'x'.repeat(30_000) } }; + } + async onProcess(event: ProcessEvent): Promise { + if (event.type !== 'external-message') return {}; + return { + addMessages: [{ participant: 'User', content: (event as { content: unknown }).content as never }], + requestInference: true, + }; + } +} +function nearCapToolResponse() { + const r = createMockResponse([ + { type: 'text', text: 'Working' }, + { type: 'tool_use', id: 'call_1', name: 'canned--fetch', input: {} }, + ], 'tool_use'); + (r as { usage: unknown }).usage = { inputTokens: 5_000, outputTokens: 5, cacheReadTokens: 195_000 }; + return r; +} + +async function heldRestart(texts: string[], options: { + fits?: boolean; + duringTool?: (internal: Internal) => unknown; + inferencePolicy?: InferencePolicy; + guardRetry?: boolean; + failsAfterTools?: boolean; +} = {}) { + const proto = ContextManager.prototype as unknown as Patchable; + const original = { compile: proto.compile, tick: proto.tick, isReady: proto.isReady }; + let fit = options.fits === true; + let committing = 0; + proto.compile = async function (this: unknown, ...args: unknown[]) { + const dryRun = (args[2] as { dryRun?: boolean } | undefined)?.dryRun === true; + const refused = dryRun ? !fit : ++committing >= 2 && !fit; + if (refused) throw overBudget(49_000); + return original.compile.apply(this, args); + }; + proto.tick = async function () {}; + proto.isReady = function () { return fit; }; + const tempDir = mkdtempSync(join(tmpdir(), 'budget-hold-restart-')); + const scripted = options.guardRetry || options.failsAfterTools; + const membrane = scripted + ? new ScriptedStreamMembrane([ + [ + createMockResponse([{ type: 'tool_use', id: 'call_1', name: 'canned--fetch', input: {} }], 'tool_use'), + options.guardRetry + ? { ...createMockResponse([{ type: 'text', text: 'refused partial' }], 'refusal'), raw: { request: {}, response: { stop_details: { category: 'test-category' } } } } as NormalizedResponse + : 'fails', + ], + ...texts.map((text) => [createMockResponse([{ type: 'text', text }])]), + ]) + : new SequentialStreamMembrane(); + if (!scripted) { + membrane.pushResponse(nearCapToolResponse()); + for (const text of texts) membrane.pushResponse(createMockResponse([{ type: 'text', text }])); + } + const framework = await AgentFramework.create({ + storePath: join(tempDir, 'store'), + membrane: membrane.asMembrane(), + agents: [options.guardRetry + ? { name: 'prime', model: 'test-model', systemPrompt: 'You are prime.', allowedTools: 'all', maxTokens: 4_000, toolResultGuard: true } + : { name: 'prime', model: 'test-model', systemPrompt: 'You are prime.', allowedTools: 'all', maxTokens: 4_000, physicalWindowTokens: 200_000 }], + modules: [new BlobToolModule(() => options.duringTool?.(internal))], + syncIntervalMs: 0, + maintenanceIntervalMs: 50, + errorPolicy: { maxRetries: 3, onInferenceError: (_error: Error, _agent: string, attempt: number) => (attempt < 3 ? { retry: true, delayMs: 0 } : { retry: false }) }, + ...(options.inferencePolicy ? { inferencePolicy: options.inferencePolicy } : {}), + }); + const internal = framework as unknown as Internal; + const traces: TraceEvent[] = []; + framework.onTrace((e) => traces.push(e)); + let looks = 0; + const hold = framework as unknown as { checkBudgetHold?: (agentName: string) => Promise }; + const checkBudgetHold = hold.checkBudgetHold?.bind(framework); + if (checkBudgetHold) hold.checkBudgetHold = (agentName: string) => { looks++; return checkBudgetHold(agentName); }; + framework.start(); + framework.getAgent('prime')!.getContextManager().addMessage('user', [{ type: 'text', text: 'fetch it' }]); + internal.pendingRequests.push({ agentName: 'prime', reason: 'external-message', source: 'test', timestamp: Date.now() } as unknown as { agentName: string }); + return { + framework, + internal, + membrane, + of: (type: string) => traces.filter((e) => e.type === type) as unknown as Array>, + looks: () => looks, + fit: () => { fit = true; }, + close: async () => { + try { + await framework.stop(); + } finally { + Object.assign(proto, original); + rmSync(tempDir, { recursive: true, force: true }); + } + }, + }; +} + +interface Internal { + budgetHoldMs: number; + budgetHoldRecheckMs: number; + budgetHolds: Map; + activeTurnTokens: Map; + pendingRequests: Array<{ agentName: string }>; + providerGateBlocked: (agentName: string) => boolean; + disposeConversationAgent: (agentName: string) => void; +} +type HeldRequest = { agentName: string; reason: string; coalescingBatch?: boolean; coalescingBatchSubjects?: string[]; coalescingSubject?: string }; +const holdOf = (t: { internal: Internal }) => + (t.internal.budgetHolds as unknown as Map).get('prime'); + +const heldRequests = (framework: AgentFramework): number | undefined => { + const agents = (framework.healthSnapshot() as { agents: Array<{ name: string; budgetHold: { heldRequests: number } | null }> }).agents; + return agents.find((a) => a.name === 'prime')?.budgetHold?.heldRequests; +}; + +async function residentTurn(options: { + maintenanceIntervalMs: number; holdMs?: number; recheckMs?: number; responses?: number; inferencePolicy?: InferencePolicy; + thinkingBinding?: { binds: boolean; dropBlock: boolean }; + context?: () => string; + storePath?: string; + failFirstStream?: () => void; + policyErrors?: Error[]; + clock?: { intervalSeconds: number }; +}) { + const tempDir = options.storePath === undefined ? mkdtempSync(join(tmpdir(), 'budget-hold-')) : undefined; + const membrane = new SequentialStreamMembrane(); + if (options.failFirstStream) { + const stream = membrane.streamYielding.bind(membrane); + let failed = false; + membrane.streamYielding = (request: NormalizedRequest, opts?: unknown): YieldingStream => { + if (failed) return stream(request, opts); + failed = true; + options.failFirstStream!(); + throw new Error('provider overloaded'); + }; + } + membrane.pushResponse(createMockResponse([{ type: 'text', text: 'Answered once it fit' }])); + for (let i = 1; i < (options.responses ?? 1); i++) { + membrane.pushResponse(createMockResponse([{ type: 'text', text: `Answered turn ${i + 1}` }])); + } + const framework = await AgentFramework.create({ + storePath: options.storePath ?? join(tempDir!, 'store'), + membrane: membrane.asMembrane(), + agents: [{ + name: 'prime', model: 'test-model', systemPrompt: 'You are prime.', + ...(options.thinkingBinding ? { thinkingBinding: options.thinkingBinding } : {}), + ...(options.clock ? { clock: options.clock } : {}), + }], + modules: [new InboxModule(options.context)], + syncIntervalMs: 0, + maintenanceIntervalMs: options.maintenanceIntervalMs, + ...(options.inferencePolicy ? { inferencePolicy: options.inferencePolicy } : {}), + errorPolicy: { maxRetries: 3, onInferenceError: (error, _agent, attempt) => { + options.policyErrors?.push(error); + return attempt < 3 ? { retry: true, delayMs: 0 } : { retry: false }; + } }, + }); + const internal = framework as unknown as Internal; + if (options.holdMs !== undefined) internal.budgetHoldMs = options.holdMs; + if (options.recheckMs !== undefined) internal.budgetHoldRecheckMs = options.recheckMs; + const traces: TraceEvent[] = []; + framework.onTrace((e) => traces.push(e)); + let looks = 0; + const hold = framework as unknown as { checkBudgetHold?: (agentName: string) => Promise }; + const checkBudgetHold = hold.checkBudgetHold?.bind(framework); + if (checkBudgetHold) hold.checkBudgetHold = (agentName: string) => { looks++; return checkBudgetHold(agentName); }; + const send = (text: string) => framework.pushEvent({ + type: 'external-message', + source: 'test', + content: [{ type: 'text', text }], + metadata: {}, + triggerInference: true, + } as unknown as ProcessEvent); + framework.start(); + const of = (type: string) => traces.filter((e) => e.type === type) as unknown as Array>; + const kinds = (types: string[]) => traces.filter((e) => types.includes(e.type)).map((e) => e.type); + return { + framework, + internal, + membrane, + send, + of, + kinds, + looks: () => looks, + close: async () => { + await framework.stop(); + if (tempDir) rmSync(tempDir, { recursive: true, force: true }); + }, + }; +} + +describe('a compile refused for budget holds its turn (#304)', () => { + it('runs the refused turn once compression makes the context fit, with no new wake', async () => { + await withBacklog({ fitAt: 40 }, async (backlog) => { + const t = await residentTurn({ maintenanceIntervalMs: 50 }); + try { + t.send('Are you there?'); + assert.ok(await pollUntil(() => t.of('inference:held').length === 1, 5_000)); + assert.equal(t.internal.providerGateBlocked('prime'), false, 'the hold leaves the provider gate open, so compression runs'); + assert.ok(await pollUntil(() => fits(backlog), 10_000), 'compression should catch up'); + const answered = await pollUntil(() => storedText(t.framework).includes('Answered once it fit'), 5_000); + assert.ok(answered, 'the refused turn should run once the context fits, with no new message to wake it'); + assert.ok(await pollUntil(() => t.of('inference:completed').length === 1, 5_000), 'and completes'); + assert.equal(t.membrane.calls.length, 1, 'one provider call: the fresh compile after the hold, never a refused one'); + + const held = t.of('inference:held'); + assert.equal(held.length, 1); + assert.equal(held[0]!.errorType, 'over_budget'); + assert.equal(held[0]!.budget, 48_568); + assert.equal(held[0]!.actual, 100_201); + assert.equal((held[0]!.until as number) - (held[0]!.since as number), 30 * 60_000, 'a fixed 30-minute bound'); + const alerts = t.of('ops:alert').filter((e) => e.kind === 'context-refusal'); + assert.equal(alerts.length, 1, 'the context-refusal alert went out when the hold started'); + assert.match(String(alerts[0]!.message), /the turn is held until \S+ while compression catches up/); + assert.deepEqual(t.of('inference:hold_ended').map((e) => e.outcome), ['fit']); + assert.deepEqual(t.of('inference:exhausted'), [], 'the turn never failed'); + assert.deepEqual( + t.kinds(['inference:started', 'inference:failed', 'inference:held', 'inference:hold_ended', 'inference:completed', 'inference:exhausted']), + ['inference:started', 'inference:failed', 'inference:held', 'inference:hold_ended', 'inference:started', 'inference:completed'], + 'the refused attempt fails, the hold starts and ends, and the turn starts again and completes', + ); + assert.ok(!storedText(t.framework).includes('[inference-failed]'), 'no failure marker is written for a held turn'); + assert.equal(t.internal.budgetHoldRecheckMs, 5_000, 'unshortened, the hold paces itself at the maintenance timer\'s default interval'); + } finally { + await t.close(); + } + }); + }); + + it('tells the agent why its turn waited, in a note stored where the released turn read it', async () => { + await withBacklog({ fitAt: 12 }, async () => { + const t = await residentTurn({ maintenanceIntervalMs: 50, responses: 2 }); + try { + t.send('Are you there?'); + assert.ok(await pollUntil(() => storedText(t.framework).includes('Answered once it fit'), 10_000)); + const note = /\[budget-hold\] Your context didn't fit its budget \(100,201 tokens against 48,568\), so this turn waited \d+ s for compression to make room\./; + assert.match(JSON.stringify(t.membrane.calls[0]), note); + const notes = stored(t.framework).filter((m) => m.text.includes('[budget-hold]')); + assert.equal(notes.length, 1, 'the note is stored once'); + assert.match(notes[0]!.text, note); + t.send('And now?'); + assert.ok(await pollUntil(() => storedText(t.framework).includes('Answered turn 2'), 5_000)); + assert.equal(t.membrane.calls.length, 2); + assert.equal(JSON.stringify(t.membrane.calls[1]).split('[budget-hold]').length - 1, 1, + 'the next turn reads the stored note as history, and gets no note of its own'); + } finally { + await t.close(); + } + }); + }); + + it('stores the note once: a restart inside the released turn compiles it where it was, with no second copy', async () => { + await withBacklog({ fitAt: 12 }, async () => { + const tempDir = mkdtempSync(join(tmpdir(), 'budget-hold-restart-')); + const membrane = new SequentialStreamMembrane(); + membrane.pushResponse(nearCapToolResponse()); + membrane.pushResponse(createMockResponse([{ type: 'text', text: 'Answered after the restart' }])); + const framework = await AgentFramework.create({ + storePath: join(tempDir, 'store'), + membrane: membrane.asMembrane(), + agents: [{ name: 'prime', model: 'test-model', systemPrompt: 'You are prime.', allowedTools: 'all', maxTokens: 4_000, physicalWindowTokens: 200_000 }], + modules: [new BlobToolModule()], + syncIntervalMs: 0, + maintenanceIntervalMs: 50, + errorPolicy: { maxRetries: 3, onInferenceError: (_error, _agent, attempt) => (attempt < 3 ? { retry: true, delayMs: 0 } : { retry: false }) }, + }); + const traces: TraceEvent[] = []; + framework.onTrace((e) => traces.push(e)); + try { + framework.start(); + framework.pushEvent({ + type: 'external-message', source: 'test', content: [{ type: 'text', text: 'fetch it' }], metadata: {}, triggerInference: true, + } as unknown as ProcessEvent); + assert.ok(await pollUntil(() => storedText(framework).includes('Answered after the restart'), 10_000)); + assert.ok(traces.some((e) => e.type === 'inference:held'), 'the turn was held first'); + assert.ok(traces.some((e) => e.type === 'inference:stream_restarted'), 'and restarted inside the turn'); + assert.equal(membrane.calls.length, 2); + const copies = (i: number) => JSON.stringify(membrane.calls[i]).split('[budget-hold]').length - 1; + assert.equal(copies(0), 1, 'the first compile after the release carries the note'); + assert.equal(copies(1), 1, 'the restart\'s compile has it once, where it was stored'); + assert.equal(stored(framework).filter((m) => m.text.includes('[budget-hold]')).length, 1); + } finally { + await framework.stop(); + rmSync(tempDir, { recursive: true, force: true }); + } + }); + }); + + it('stores the note as the host\'s own row, which leaves the turn\'s routing as it was', async () => { + await withBacklog({ fitAt: 12 }, async () => { + const t = await residentTurn({ maintenanceIntervalMs: 50 }); + type Routing = { triggerChannel?: unknown; locus?: string; engaged: Set; pin?: string; destination: string | null; arrivals: unknown[] }; + const routingNow = () => { + const r = (t.framework as unknown as { turnRouting: Map }).turnRouting.get('prime'); + return r && { + triggerChannel: r.triggerChannel, locus: r.locus, engaged: [...r.engaged].sort(), pin: r.pin, + destination: r.destination, arrivals: r.arrivals.length, + }; + }; + const atStart: unknown[] = []; + const afterCompile: unknown[] = []; + t.framework.onTrace((e) => { + if (e.type === 'inference:started') atStart.push(routingNow()); + if (e.type === 'inference:completed') afterCompile.push(routingNow()); + }); + try { + await (t.framework as unknown as { handleMcplChannelIncoming: (e: unknown) => Promise }).handleMcplChannelIncoming({ + type: 'mcpl:channel-incoming', serverId: 'discord', channelId: 'discord:guild:chanA', messageId: 'm-1', + author: { id: 'U1', name: 'owner' }, content: [{ type: 'text', text: 'Are you there?' }], + timestamp: new Date().toISOString(), metadata: {}, triggerInference: true, + }); + assert.ok(await pollUntil(() => t.of('inference:completed').length === 1, 10_000)); + assert.equal(t.of('inference:held').length, 1); + assert.ok(JSON.stringify(t.membrane.calls[0]).includes('[budget-hold]'), 'the released compile carried the note'); + const rows = (t.framework.getAgent('prime')!.getContextManager().queryMessages({}).messages as unknown as Array<{ metadata?: Record }>) + .filter((m) => JSON.stringify(m).includes('[budget-hold]')); + assert.equal(rows.length, 1); + assert.equal(rows[0]!.metadata?.system, true, 'stored as the host\'s own row'); + assert.equal(rows[0]!.metadata?.kind, 'context-injection'); + assert.equal(afterCompile.length, 1); + const released = atStart[atStart.length - 1] as { triggerChannel?: unknown }; + assert.deepEqual(released.triggerChannel, { serverId: 'discord', channelId: 'discord:guild:chanA' }, 'the released turn starts from the message\'s channel'); + assert.deepEqual(afterCompile[0], released, 'and the stored note leaves its routing as it was'); + } finally { + await t.close(); + } + }); + }); + + it('takes later wakes into the held turn, for one compile', async () => { + await withBacklog({}, async (backlog) => { + const t = await residentTurn({ maintenanceIntervalMs: 50 }); + try { + t.send('First message'); + assert.ok(await pollUntil(() => t.of('inference:held').length === 1, 5_000)); + t.send('Second message, while held'); + assert.ok(await pollUntil(() => heldRequests(t.framework) === 2, 5_000), 'the second wake joins the hold'); + assert.equal(t.membrane.calls.length, 0, 'no compile while held, so no provider call'); + assert.equal(t.of('inference:failed').length, 1, 'only the first refusal: the second wake never compiled while held'); + backlog.fit = true; + assert.ok(await pollUntil(() => storedText(t.framework).includes('Answered once it fit'), 5_000)); + assert.equal(t.membrane.calls.length, 1, 'both wakes ran as one turn'); + const request = JSON.stringify(t.membrane.calls[0]); + assert.ok(request.includes('First message') && request.includes('Second message, while held')); + assert.equal(t.of('inference:held').length, 1, 'the second wake joined the hold, and started none'); + } finally { + await t.close(); + } + }); + }); + + it('keeps today\'s failure when nothing is pending: compression cannot change it', async () => { + await withBacklog({ readyAt: 0 }, async () => { + await errorLines(async (lines) => { + const t = await residentTurn({ maintenanceIntervalMs: 50 }); + try { + t.send('Are you there?'); + assert.ok(await pollUntil(() => t.of('inference:exhausted').length === 1, 5_000), 'the turn fails, as before the hold'); + assert.deepEqual(t.of('inference:held'), []); + assert.equal(t.of('inference:exhausted')[0]!.errorType, 'over_budget'); + assert.ok(storedText(t.framework).includes('[inference-failed]'), 'the agent gets the failure marker'); + assert.ok(await pollUntil(() => lines.some((l) => l.includes('drain kicked for prime')), 3_000)); + const kicks = lines.filter((l) => l.includes('drain kicked for prime')); + assert.ok(kicks.every((l) => l.startsWith('[inference-failed] drain kicked for prime (OverBudget breaker, ')), + `the breaker's kick logs as the breaker's: ${kicks.join(' | ')}`); + } finally { + await t.close(); + } + }); + }); + }); + + it('ends the hold when compression has nothing left to do and the context still does not fit', async () => { + await withBacklog({ readyAt: 20 }, async () => { + const t = await residentTurn({ maintenanceIntervalMs: 50 }); + try { + t.send('Are you there?'); + assert.ok(await pollUntil(() => t.of('inference:exhausted').length === 1, 5_000)); + assert.deepEqual(t.of('inference:hold_ended').map((e) => e.outcome), ['nothing_pending']); + const exhausted = t.of('inference:exhausted')[0]!; + assert.equal(exhausted.errorType, 'over_budget'); + assert.match(String(exhausted.error), /the turn was first held \d+ s ago; compression had nothing left to do and it still didn't fit/); + assert.equal(t.membrane.calls.length, 0); + } finally { + await t.close(); + } + }); + }); + + it('ends the hold at its deadline, and the turn fails as a refused compile did', async () => { + await withBacklog({}, async (backlog) => { + const policyErrors: Error[] = []; + const t = await residentTurn({ maintenanceIntervalMs: 50, holdMs: 400, policyErrors }); + try { + t.send('Are you there?'); + assert.ok(await pollUntil(() => t.of('inference:exhausted').length === 1, 5_000)); + assert.deepEqual(t.of('inference:hold_ended').map((e) => e.outcome), ['deadline']); + assert.match(String(t.of('inference:exhausted')[0]!.error), /the turn was first held \d+ s ago; the hold's 1 s bound passed and it still didn't fit/); + assert.equal(t.membrane.calls.length, 0); + const last = policyErrors.at(-1) as Error & { budget?: number; actual?: number }; + assert.equal(last, backlog.thrown.at(-1), 'the error policy gets the refusal itself'); + assert.equal(last.budget, 48_568, 'with its figures'); + assert.match(last.message, /the turn was first held/); + } finally { + await t.close(); + } + }); + }); + + it('holds a later turn afresh after a hold that ended in failure, and after one that ran', async () => { + await withBacklog({}, async (backlog) => { + const t = await residentTurn({ maintenanceIntervalMs: 50, holdMs: 400 }); + try { + t.send('First'); + assert.ok(await pollUntil(() => t.of('inference:exhausted').length === 1, 5_000)); + t.send('Second'); + assert.ok(await pollUntil(() => t.of('inference:held').length === 2, 5_000), 'a failed turn\'s spent hold does not pass to the next turn'); + backlog.fit = true; + assert.ok(await pollUntil(() => storedText(t.framework).includes('Answered once it fit'), 5_000)); + backlog.fit = false; + const before = Date.now(); + t.send('Third'); + assert.ok(await pollUntil(() => t.of('inference:held').length === 3, 5_000)); + const third = t.of('inference:held')[2]!; + assert.ok((third.since as number) >= before, 'its own start, not the released turn\'s'); + assert.ok((third.until as number) >= before + 400); + } finally { + await t.close(); + } + }); + }); + + it('runs the turn when the check fails for another reason, and the turn\'s own compile meets it', async () => { + await withBacklog({}, async (backlog) => { + const t = await residentTurn({ maintenanceIntervalMs: 50 }); + try { + t.send('Are you there?'); + assert.ok(await pollUntil(() => t.of('inference:held').length === 1, 5_000)); + backlog.previewError = new Error('store read failed'); + backlog.fit = true; + assert.ok(await pollUntil(() => storedText(t.framework).includes('Answered once it fit'), 5_000)); + assert.deepEqual(t.of('inference:hold_ended').map((e) => e.outcome), ['check_failed']); + assert.equal(backlog.previewError, undefined, 'the preview met the error'); + assert.ok(JSON.stringify(t.membrane.calls[0]).includes('[budget-hold]'), 'the turn that ran says why it waited'); + } finally { + await t.close(); + } + }); + }); + + it('paces compression with the maintenance timer off: the drain kick runs again until it fits', async () => { + await withBacklog({ fitAt: 20 }, async (backlog) => { + const t = await residentTurn({ maintenanceIntervalMs: 0, recheckMs: 50 }); + try { + t.send('Are you there?'); + assert.ok(await pollUntil(() => storedText(t.framework).includes('Answered once it fit'), 5_000), + 'the kick goes on past its 8 ticks while compression is pending'); + assert.ok(backlog.ticks >= 20); + assert.deepEqual(t.of('inference:hold_ended').map((e) => e.outcome), ['fit']); + } finally { + await t.close(); + } + }); + }); + + it('kicks the drain at once when the hold starts, and logs the kick as the hold\'s', async () => { + await withBacklog({ fitAt: 8 }, async () => { + await errorLines(async (lines) => { + const t = await residentTurn({ maintenanceIntervalMs: 0, recheckMs: 60_000 }); + try { + t.send('Are you there?'); + assert.ok(await pollUntil(() => storedText(t.framework).includes('Answered once it fit'), 3_000)); + assert.deepEqual(t.of('inference:hold_ended').map((e) => e.outcome), ['fit']); + const kicks = lines.filter((l) => l.includes('drain kicked for prime')); + assert.deepEqual(kicks, ['[budget-hold] drain kicked for prime (budget hold, 8 ticks)'], 'one kick, logged as the hold\'s'); + } finally { + await t.close(); + } + }); + }); + }); + + it('with the maintenance timer off, a backlog that never moves costs a kick an interval, not a loop', async () => { + await withBacklog({}, async (backlog) => { + const t = await residentTurn({ maintenanceIntervalMs: 0, recheckMs: 100 }); + try { + t.send('Are you there?'); + assert.ok(await pollUntil(() => t.of('inference:held').length === 1, 5_000)); + const since = Date.now(); + assert.ok(await pollUntil(() => backlog.ticks >= 16, 5_000), 'it does kick again'); + await new Promise((r) => setTimeout(r, 300)); + const intervals = Math.ceil((Date.now() - since) / 100) + 1; + assert.ok(backlog.ticks <= 8 * (intervals + 1), `${backlog.ticks} ticks in ${intervals} intervals`); + assert.ok(backlog.previews <= intervals + 1, `${backlog.previews} checks in ${intervals} intervals`); + } finally { + await t.close(); + } + }); + }); + + it('with the maintenance timer off and the provider gate closed, looks again an interval later', async () => { + await withBacklog({}, async (backlog) => { + const t = await residentTurn({ maintenanceIntervalMs: 0, recheckMs: 50 }); + const open = t.internal.providerGateBlocked.bind(t.framework); + let blocked = true; + t.internal.providerGateBlocked = (name: string) => (name === 'prime' && blocked) || open(name); + try { + t.send('Are you there?'); + assert.ok(await pollUntil(() => t.of('inference:held').length === 1, 5_000)); + await new Promise((r) => setTimeout(r, 200)); + assert.equal(backlog.ticks, 0, 'no kick while the gate is closed: its calls would park there'); + backlog.fit = true; + assert.ok(await pollUntil(() => t.of('inference:hold_ended').length === 1, 3_000), + 'the hold looked again an interval later, with the gate still closed, and released the turn'); + assert.equal(backlog.ticks, 0); + blocked = false; + assert.ok(await pollUntil(() => storedText(t.framework).includes('Answered once it fit'), 3_000)); + } finally { + await t.close(); + } + }); + }); + + it('keeps the held turn alive: the host reads it as undrained, its status shows the hold, and a surgery is refused naming it', async () => { + await withBacklog({}, async () => { + const t = await residentTurn({ maintenanceIntervalMs: 3_600_000 }); + try { + t.send('Are you there?'); + assert.ok(await pollUntil(() => t.of('inference:held').length === 1, 5_000)); + const status = t.framework.getHostModeStatus(); + assert.equal(status.drained, false, 'a held turn is a turn alive'); + assert.deepEqual(status.activeTurnAgents, ['prime']); + const agents = (t.framework.healthSnapshot() as { agents: Array<{ name: string; budgetHold: Record | null }> }).agents; + const hold = agents.find((a) => a.name === 'prime')!.budgetHold; + assert.ok(hold, 'the status names the hold'); + assert.equal(hold!.errorType, 'over_budget'); + assert.equal(hold!.heldRequests, 1); + assert.equal(status.gatedRequests, 0, 'the turn\'s own retry is no gated wake'); + t.send('Joined while held'); + assert.ok(await pollUntil(() => heldRequests(t.framework) === 2, 5_000)); + assert.equal(t.framework.getHostModeStatus().gatedRequests, 1, 'a wake that joined the hold waits to start'); + await assert.rejects(t.framework.switchBranch('main'), (e: Error) => { + assert.equal((e as Error & { code?: string }).code, 'agent-busy'); + assert.match(e.message, /prime is idle\+turn-alive \(a budget hold keeps its turn to start again once its context fits, or at \S+ at the latest: Compile plan would exceed hard budget/); + assert.match(e.message, /end it with quiesce with abandon: true/); + return true; + }); + assert.equal(heldRequests(t.framework), 2, 'the turn is still held'); + } finally { + await t.close(); + } + }); + }); + + it('refuses a rollback while a turn is held, so the turn never runs on history rewound under it', async () => { + await withBacklog({ fit: true }, async (backlog) => { + const t = await residentTurn({ maintenanceIntervalMs: 50, responses: 2 }); + try { + t.send('First'); + assert.ok(await pollUntil(() => storedText(t.framework).includes('Answered once it fit'), 5_000)); + const msgs = t.framework.getAgent('prime')!.getContextManager().queryMessages({}).messages as unknown as Array<{ id: unknown }>; + const answer = String(msgs[msgs.length - 1]!.id); + backlog.fit = false; + t.send('Are you there?'); + assert.ok(await pollUntil(() => t.of('inference:held').length === 1 && backlog.previews >= 1, 5_000)); + await assert.rejects(t.framework.rollbackToMessage('prime', { messageId: answer }), (e: Error) => { + assert.equal((e as Error & { code?: string }).code, 'agent-busy'); + assert.match(e.message, /prime is idle\+turn-alive \(a budget hold/); + return true; + }); + assert.ok(storedText(t.framework).includes('Are you there?'), 'the held turn\'s message is still on the branch'); + backlog.fit = true; + t.send('Still there?'); + assert.ok(await pollUntil(() => storedText(t.framework).includes('Answered turn 2'), 5_000)); + assert.ok(JSON.stringify(t.membrane.calls[1]).includes('Are you there?'), 'the released turn reads the message it was held for'); + } finally { + await t.close(); + } + }); + }); + + it('stops the hold\'s drain kick once stop() begins, and starts no kick after it', async () => { + await withBacklog({}, async (backlog) => { + await errorLines(async (lines) => { + const t = await residentTurn({ maintenanceIntervalMs: 3_600_000 }); + const fw = t.framework as unknown as { + maintenancePass: unknown; + overBudgetDrainAfter: Map; + kickOverBudgetDrain: (name: string, agent: unknown, after?: () => void, by?: 'breaker' | 'hold') => string; + }; + const proto = ContextManager.prototype as unknown as Patchable; + let letGo = () => {}; + let stopping: Promise | undefined; + try { + assert.ok(await pollUntil(() => fw.maintenancePass === null, 3_000), 'the start pass is done'); + const gate = new Promise((resolve) => { letGo = resolve; }); + proto.tick = async function () { backlog.ticks++; await gate; }; + const before = backlog.ticks; + t.send('Are you there?'); + assert.ok(await pollUntil(() => t.of('inference:held').length === 1 && backlog.ticks === before + 1, 5_000), + 'the hold\'s kick is in its first tick'); + let stopped = false; + stopping = t.close().then(() => { stopped = true; }); + await new Promise((r) => setTimeout(r, 200)); + assert.equal(stopped, false, 'stop() waits for the tick in flight'); + const waiting = fw.overBudgetDrainAfter.get('prime')?.length ?? 0; + assert.equal(fw.kickOverBudgetDrain('prime', t.framework.getAgent('prime'), () => {}, 'hold'), + 'not kicked: the framework is stopping'); + assert.equal(fw.overBudgetDrainAfter.get('prime')?.length ?? 0, waiting, 'and keeps no callback'); + letGo(); + await stopping; + assert.equal(backlog.ticks, before + 1, 'the kick ticked no more once stop() began'); + const kicks = lines.filter((l) => l.includes('drain kicked for prime')); + assert.deepEqual(kicks, ['[budget-hold] drain kicked for prime (budget hold, 1 ticks, stopped as the framework stops)']); + } finally { + letGo(); + await (stopping ?? t.close()); + } + }); + }); + }); + + it('joins a drain kick already in flight: no second kick, and the hold looks when it ends', async () => { + await withBacklog({}, async (backlog) => { + const t = await residentTurn({ maintenanceIntervalMs: 3_600_000 }); + const fw = t.framework as unknown as { + maintenancePass: unknown; + overBudgetDrains: Set>; + overBudgetDrainAfter: Map; + kickOverBudgetDrain: (name: string, agent: unknown, after?: () => void, by?: 'breaker' | 'hold') => string; + }; + const proto = ContextManager.prototype as unknown as Patchable; + let letGo = () => {}; + try { + assert.ok(await pollUntil(() => fw.maintenancePass === null, 3_000), 'the start pass is done'); + const gate = new Promise((resolve) => { letGo = resolve; }); + proto.tick = async function () { backlog.ticks++; await gate; }; + const before = backlog.ticks; + t.send('Are you there?'); + assert.ok(await pollUntil(() => t.of('inference:held').length === 1 && backlog.ticks === before + 1, 5_000), + 'the hold\'s kick is in its first tick'); + assert.equal(fw.overBudgetDrains.size, 1); + let joined = 0; + assert.equal(fw.kickOverBudgetDrain('prime', t.framework.getAgent('prime'), () => { joined++; }, 'hold'), + 'already draining'); + assert.equal(fw.overBudgetDrains.size, 1, 'no second kick starts'); + assert.equal(fw.overBudgetDrainAfter.get('prime')?.length, 2, 'its callback waits beside the hold\'s own'); + const asked = t.looks(); + letGo(); + assert.ok(await pollUntil(() => joined === 1 && !fw.overBudgetDrainAfter.has('prime'), 3_000), + 'both callbacks ran when the kick ended'); + assert.ok(t.looks() > asked, 'the hold looked when the kick ended'); + assert.equal(backlog.ticks, before + 8, 'one kick\'s eight ticks'); + assert.ok(await pollUntil(() => fw.overBudgetDrains.size === 0, 1_000), 'and the one kick has left the drains'); + } finally { + letGo(); + await t.close(); + } + }); + }); + + it('keeps holding where a dry compile would resolve but the preview says the context does not fit', async () => { + await withBacklog({ dryRunResolves: true }, async (backlog) => { + const t = await residentTurn({ maintenanceIntervalMs: 50 }); + try { + t.send('Are you there?'); + assert.ok(await pollUntil(() => t.of('inference:held').length === 1 && backlog.previews >= 3, 5_000)); + assert.deepEqual(t.of('inference:hold_ended'), [], 'held through each look'); + assert.equal(backlog.compiles, 1, 'and never released into the refusal'); + backlog.fit = true; + assert.ok(await pollUntil(() => storedText(t.framework).includes('Answered once it fit'), 5_000)); + assert.deepEqual(t.of('inference:hold_ended').map((e) => e.outcome), ['fit']); + } finally { + await t.close(); + } + }); + }); + + it('asks for its verdict with the thinking binding the turn\'s compile carries, from the preview and from a dry compile', async () => { + for (const previewless of [false, true]) { + await withBacklog({ fitAt: 12, previewless }, async (backlog) => { + const t = await residentTurn({ maintenanceIntervalMs: 50, thinkingBinding: { binds: true, dropBlock: true } }); + try { + t.send('Are you there?'); + assert.ok(await pollUntil(() => storedText(t.framework).includes('Answered once it fit'), 10_000)); + const binding = { binds: true, dropBlock: true, participant: 'prime' }; + assert.deepEqual(backlog.bindings.compile, [binding, binding], 'the refused compile and the released one'); + const asked = previewless ? backlog.bindings.dryRun : backlog.bindings.preview; + assert.ok(asked.length >= 1); + for (const given of asked) assert.deepEqual(given, binding); + assert.deepEqual(t.of('inference:hold_ended').map((e) => e.outcome), ['fit'], 'the verdict was a fit, not a failed check'); + } finally { + await t.close(); + } + }); + } + }); + + it('falls back to a dry compile for a strategy with no preview, one at a time, and lands a fit', async () => { + await withBacklog({ previewless: true, dryRunMs: 300, fitAt: 40, readyAt: 40 }, async (backlog) => { + const t = await residentTurn({ maintenanceIntervalMs: 50, holdMs: 6_000 }); + try { + t.send('Are you there?'); + assert.ok(await pollUntil(() => t.of('inference:held').length === 1, 5_000)); + assert.ok(await pollUntil(() => storedText(t.framework).includes('Answered once it fit'), 5_000), + 'the turn runs soon after the fit, not at the deadline'); + assert.deepEqual(t.of('inference:hold_ended').map((e) => e.outcome), ['fit'], + 'a refusal decided before compression finished is not taken for nothing pending'); + assert.ok(backlog.dryRuns >= 1 && backlog.previews >= 1, 'the preview had nothing to say, and a dry compile decided'); + assert.equal(backlog.mostDryRunsRunning, 1, 'one dry compile at a time'); + assert.ok(backlog.dryRunsAcrossFit >= 1, 'a dry compile began before the fit and ended after it, which is what this test is for'); + } finally { + await t.close(); + } + }); + }); + + it('on the fallback path, looks again at the next pass after a message a dry compile straddled', async () => { + await withBacklog({ previewless: true, stalled: true }, async (backlog) => { + const t = await residentTurn({ maintenanceIntervalMs: 50 }); + try { + t.send('Are you there?'); + assert.ok(await pollUntil(() => t.of('inference:held').length === 1 && backlog.dryRuns >= 1 && backlog.dryRunsRunning === 0, 5_000)); + let release!: () => void; + backlog.dryRunGate = new Promise((resolve) => { release = resolve; }); + const dryRuns = backlog.dryRuns; + t.send('A message while held'); + assert.ok(await pollUntil(() => backlog.dryRuns === dryRuns + 1, 2_000)); + backlog.fit = true; + t.send('A second message while held'); + assert.ok(await pollUntil(() => storedText(t.framework).includes('A second message while held'), 2_000)); + backlog.dryRunGate = undefined; + release(); + assert.ok(await pollUntil(() => storedText(t.framework).includes('Answered once it fit'), 3_000), + 'the stale refusal stood only until the next pass, which the second message had given something new to look at'); + assert.equal(backlog.mostDryRunsRunning, 1); + } finally { + await t.close(); + } + }); + }); + + it('on the fallback path, sets aside a refusal that compression coming back made stale', async () => { + await withBacklog({ previewless: true, noProgress: true, alwaysPending: true, readyAt: 0 }, async (backlog) => { + const t = await residentTurn({ maintenanceIntervalMs: 50 }); + try { + t.send('Are you there?'); + assert.ok(await pollUntil(() => t.of('inference:held').length === 1 && backlog.dryRuns >= 1 && backlog.dryRunsRunning === 0, 5_000)); + let release!: () => void; + backlog.dryRunGate = new Promise((resolve) => { release = resolve; }); + const dryRuns = backlog.dryRuns; + backlog.alwaysPending = false; + assert.ok(await pollUntil(() => backlog.dryRuns === dryRuns + 1, 2_000)); + backlog.alwaysPending = true; + backlog.dryRunGate = undefined; + release(); + await new Promise((r) => setTimeout(r, 300)); + assert.deepEqual(t.of('inference:hold_ended'), [], 'the stale refusal did not end the hold as one with nothing pending'); + assert.deepEqual(t.of('inference:exhausted'), []); + backlog.fit = true; + t.send('Still there?'); + assert.ok(await pollUntil(() => storedText(t.framework).includes('Answered once it fit'), 3_000)); + assert.deepEqual(t.of('inference:hold_ended').map((e) => e.outcome), ['fit']); + } finally { + await t.close(); + } + }); + }); + + it('on the fallback path, holds on when compression comes back during a dry compile that its progress does not show', async () => { + await withBacklog({ previewless: true, stalled: true, alwaysPending: true, readyAt: 0 }, async (backlog) => { + const t = await residentTurn({ maintenanceIntervalMs: 50 }); + try { + t.send('Are you there?'); + assert.ok(await pollUntil(() => t.of('inference:held').length === 1 && backlog.dryRuns >= 1 && backlog.dryRunsRunning === 0, 5_000)); + let release!: () => void; + backlog.dryRunGate = new Promise((resolve) => { release = resolve; }); + const dryRuns = backlog.dryRuns; + backlog.alwaysPending = false; + t.send('Still there?'); + assert.ok(await pollUntil(() => backlog.dryRuns === dryRuns + 1, 2_000)); + backlog.alwaysPending = true; + backlog.dryRunGate = undefined; + release(); + await new Promise((r) => setTimeout(r, 300)); + assert.deepEqual(t.of('inference:hold_ended'), [], 'the refusal met compression with work again, so the hold goes on'); + assert.deepEqual(t.of('inference:exhausted'), []); + backlog.fit = true; + t.send('And now?'); + assert.ok(await pollUntil(() => storedText(t.framework).includes('Answered once it fit'), 3_000)); + assert.deepEqual(t.of('inference:hold_ended').map((e) => e.outcome), ['fit']); + } finally { + await t.close(); + } + }); + }); + + it('tells a turn that fits after its hold\'s deadline why it waited', async () => { + await withBacklog({}, async (backlog) => { + const t = await residentTurn({ maintenanceIntervalMs: 3_600_000, holdMs: 400 }); + try { + t.send('Are you there?'); + assert.ok(await pollUntil(() => t.of('inference:held').length === 1, 5_000)); + assert.ok(await pollUntil(() => backlog.previews >= 1, 2_000)); + backlog.fit = true; + assert.ok(await pollUntil(() => storedText(t.framework).includes('Answered once it fit'), 3_000)); + assert.deepEqual(t.of('inference:hold_ended').map((e) => e.outcome), ['deadline']); + assert.match(JSON.stringify(t.membrane.calls[0]), /\[budget-hold\][^"]*so this turn waited \d+ s for compression to make room\./); + assert.deepEqual(t.of('inference:exhausted'), []); + } finally { + await t.close(); + } + }); + }); + + it('refuses an undo while a turn is held, naming the hold', async () => { + await withBacklog({ fit: true }, async (backlog) => { + const t = await residentTurn({ maintenanceIntervalMs: 3_600_000 }); + try { + t.send('A turn that fits'); + assert.ok(await pollUntil(() => storedText(t.framework).includes('Answered once it fit'), 5_000)); + backlog.fit = false; + t.send('Are you there?'); + assert.ok(await pollUntil(() => t.of('inference:held').length === 1, 5_000)); + assert.throws(() => t.framework.undoLastTurn('prime'), (e: Error) => { + assert.equal((e as Error & { code?: string }).code, 'agent-busy'); + assert.match(e.message, /Cannot undo while prime is idle\+turn-alive \(a budget hold/); + return true; + }); + assert.equal(t.framework.getUndoRedoState('prime').undoDepth, 2, 'the earlier turn and the held one are there to undo once it ends'); + assert.ok(storedText(t.framework).includes('Are you there?')); + } finally { + await t.close(); + } + }); + }); + + it('makes one held turn one undo step, which keeps the inputs that joined it', async () => { + await withBacklog({}, async (backlog) => { + const t = await residentTurn({ maintenanceIntervalMs: 50 }); + try { + t.send('First message'); + assert.ok(await pollUntil(() => t.of('inference:held').length === 1, 5_000)); + t.send('Second message, while held'); + assert.ok(await pollUntil(() => heldRequests(t.framework) === 2, 5_000)); + backlog.fit = true; + assert.ok(await turnOver(t)); + assert.ok(storedText(t.framework).includes('Answered once it fit')); + assert.equal(t.framework.getUndoRedoState('prime').undoDepth, 1, 'the refused start left no undo step of its own'); + assert.equal(t.framework.undoLastTurn('prime').undone, true); + const text = storedText(t.framework); + assert.ok(!text.includes('Answered once it fit'), 'the undo takes the reply'); + assert.ok(text.includes('First message') && text.includes('Second message, while held'), 'and keeps both inputs'); + } finally { + await t.close(); + } + }); + }); + + it('makes a turn held on its error policy\'s retry one undo step too', async () => { + await withBacklog({ fit: true }, async (backlog) => { + const t = await residentTurn({ maintenanceIntervalMs: 50, failFirstStream: () => { backlog.fit = false; } }); + try { + t.send('First message'); + assert.ok(await pollUntil(() => t.of('inference:held').length === 1, 5_000), 'the retry\'s compile is refused and held'); + t.send('Second message, while held'); + assert.ok(await pollUntil(() => heldRequests(t.framework) === 2, 5_000)); + backlog.fit = true; + assert.ok(await turnOver(t)); + assert.ok(storedText(t.framework).includes('Answered once it fit')); + assert.equal(t.framework.getUndoRedoState('prime').undoDepth, 1, 'one undo step for the turn'); + assert.equal(t.framework.undoLastTurn('prime').undone, true); + const text = storedText(t.framework); + assert.ok(text.includes('First message') && !text.includes('Answered once it fit'), 'which takes the reply and leaves the input'); + assert.ok(!text.includes('Second message, while held'), 'and takes the message that arrived while the retry waited'); + } finally { + await t.close(); + } + }); + }); + + it('leaves a held turn that is stopped its own undo step, so an undo takes nothing from the turn before', async () => { + await withBacklog({ fit: true }, async (backlog) => { + const t = await residentTurn({ maintenanceIntervalMs: 3_600_000, responses: 2 }); + try { + t.send('First message'); + assert.ok(await turnOver(t)); + backlog.fit = false; + t.send('Second message'); + assert.ok(await pollUntil(() => t.of('inference:held').length === 1, 5_000)); + assert.equal(t.framework.abortInference('prime'), true); + assert.ok(await pollUntil(() => t.of('inference:aborted').length === 1, 3_000)); + assert.equal(t.framework.getUndoRedoState('prime').undoDepth, 2, 'the turn before and the stopped one'); + assert.equal(t.framework.undoLastTurn('prime').undone, true); + const text = storedText(t.framework); + assert.ok(text.includes('Answered once it fit') && text.includes('Second message'), 'the undo takes the stopped turn alone'); + } finally { + await t.close(); + } + }); + }); + + it('keeps the checkpoint of a turn held on its error policy\'s retry after its rounds ran, so its undo takes them', async () => { + const t = await heldRestart(['Answered once it fit'], { failsAfterTools: true }); + try { + assert.ok(await pollUntil(() => t.of('inference:held').length === 1, 5_000), 'the retry after the failed stream is refused and held'); + assert.ok(toolBlocks(t.framework) >= 2, 'the turn\'s tool round is in its history'); + t.framework.getAgent('prime')!.getContextManager().addMessage('user', [{ type: 'text', text: 'Added while held' }]); + t.fit(); + assert.ok(await turnOver(t)); + assert.ok(storedText(t.framework).includes('Answered once it fit')); + assert.equal(t.framework.getUndoRedoState('prime').undoDepth, 1, 'one undo step for the turn'); + assert.equal(t.framework.undoLastTurn('prime').undone, true); + assert.equal(toolBlocks(t.framework), 0, 'the undo takes the rounds the turn ran before its hold'); + const text = storedText(t.framework); + assert.ok(text.includes('fetch it') && !text.includes('Answered once it fit') && !text.includes('Added while held'), + 'and the reply and what arrived while the retry waited'); + } finally { + await t.close(); + } + }); + + it('keeps the turn\'s one undo step when its tool-result guard\'s retry is held', async () => { + const t = await heldRestart(['Answered once it fit'], { guardRetry: true }); + try { + assert.ok(await pollUntil(() => t.of('inference:held').length === 1, 5_000), 'the guard\'s retry is refused and held'); + assert.ok(toolBlocks(t.framework) >= 2, 'the round the guard\'s retry follows is in the turn\'s history'); + t.fit(); + assert.ok(await turnOver(t)); + assert.ok(storedText(t.framework).includes('Answered once it fit')); + assert.equal(t.framework.getUndoRedoState('prime').undoDepth, 1, 'the turn\'s first start recorded its one undo step, and the hold gave none of it back'); + assert.equal(t.framework.undoLastTurn('prime').undone, true); + assert.equal(toolBlocks(t.framework), 0, 'the undo takes the tool round the guard\'s retry followed'); + } finally { + await t.close(); + } + }); + + it('gives up the undo step of a start that reaches its compile while the hold waits', async () => { + await withBacklog({}, async (backlog) => { + const t = await residentTurn({ maintenanceIntervalMs: 50 }); + try { + t.send('First message'); + assert.ok(await pollUntil(() => t.of('inference:held').length === 1, 5_000)); + assert.ok(await pollUntil(() => !t.internal.activeTurnTokens.has('prime'), 3_000)); + const fw = t.framework as unknown as { startAgentStream: (agent: unknown, trigger: unknown, attempt: number) => Promise }; + await fw.startAgentStream(t.framework.getAgent('prime'), { agentName: 'prime', reason: 'test-start', source: 'test', timestamp: Date.now() }, 0); + assert.equal(t.of('inference:held').length, 1, 'it joined the hold rather than starting another'); + backlog.fit = true; + assert.ok(await pollUntil(() => storedText(t.framework).includes('Answered once it fit'), 5_000)); + assert.equal(t.framework.getUndoRedoState('prime').undoDepth, 1, 'the held turn is still one undo step'); + } finally { + await t.close(); + } + }); + }); + + it('holds a released turn again with its first refusal\'s start and deadline when its own compile still refuses', async () => { + await withBacklog({ fit: true, alwaysPending: true, refuseCompiles: 3 }, async () => { + const t = await residentTurn({ maintenanceIntervalMs: 50 }); + try { + t.send('Are you there?'); + assert.ok(await pollUntil(() => storedText(t.framework).includes('Answered once it fit'), 5_000)); + const held = t.of('inference:held'); + assert.equal(held.length, 3, 'each refusal of the released turn held it again'); + assert.ok(held.every((e) => e.since === held[0]!.since), 'with the first refusal\'s start'); + assert.ok(held.every((e) => e.until === held[0]!.until), 'and its deadline'); + assert.deepEqual(t.of('inference:exhausted'), []); + } finally { + await t.close(); + } + }); + }); + + it('does not release a turn its compile refused again until compression moves or the store changes', async () => { + await withBacklog({ fit: true, alwaysPending: true, refuseCompiles: Infinity, stalled: true }, async (backlog) => { + const t = await residentTurn({ maintenanceIntervalMs: 50 }); + try { + t.send('Are you there?'); + assert.ok(await pollUntil(() => t.of('inference:held').length === 2, 5_000), 'released once, refused, and held again'); + const compiles = backlog.compiles; + const asked = t.looks(); + await new Promise((r) => setTimeout(r, 600)); + assert.ok(t.looks() > asked, 'passes asked the hold to look'); + assert.equal(t.of('inference:held').length, 2, 'no release on the same evidence, pass after pass'); + assert.equal(backlog.compiles, compiles, 'and no compile refused for it'); + t.send('A message while held'); + assert.ok(await pollUntil(() => t.of('inference:held').length === 3, 3_000), + 'a message is new evidence: the hold looks, releases, and holds again'); + } finally { + await t.close(); + } + }); + }); + + it('looks again only once something that could make the context fit has changed', async () => { + await withBacklog({ stalled: true }, async (backlog) => { + const t = await residentTurn({ maintenanceIntervalMs: 50 }); + try { + t.send('Are you there?'); + assert.ok(await pollUntil(() => t.of('inference:held').length === 1 && backlog.previews >= 1, 5_000)); + let previews = backlog.previews; + let asked = t.looks(); + await new Promise((r) => setTimeout(r, 500)); + assert.ok(t.looks() > asked, 'passes asked the hold to look'); + assert.equal(backlog.previews, previews, 'and it took no verdict while nothing changed'); + t.framework.updateAgentRuntimeSettings('prime', { contextBudgetTokens: 200_000 }); + previews = backlog.previews; + assert.ok(await pollUntil(() => backlog.previews > previews, 2_000), 'it looks once the budget changes'); + previews = backlog.previews; + asked = t.looks(); + await new Promise((r) => setTimeout(r, 300)); + assert.ok(t.looks() > asked); + assert.equal(backlog.previews, previews); + backlog.stalled = false; + assert.ok(await pollUntil(() => backlog.previews > previews, 2_000), 'and once compression moves'); + } finally { + await t.close(); + } + }); + }); + + it('does not release a re-refused turn again on what its own start stored, an injection whose content changed included', async () => { + await withBacklog({ fit: true, alwaysPending: true, refuseCompiles: Infinity, stalled: true }, async (backlog) => { + let starts = 0; + const t = await residentTurn({ maintenanceIntervalMs: 50, context: () => `status at start ${++starts}` }); + try { + t.send('Are you there?'); + assert.ok(await pollUntil(() => t.of('inference:held').length === 2, 5_000), 'released once, refused, and held again'); + const stored = (t.framework.getAgent('prime')!.getContextManager().queryMessages({}).messages as unknown as Array<{ content: Array<{ text?: string }> }>) + .filter((m) => (m.content[0]?.text ?? '').startsWith('status at start')); + assert.equal(stored.length, 2, 'each start stored the injection, changed'); + const compiles = backlog.compiles; + await new Promise((r) => setTimeout(r, 600)); + assert.equal(t.of('inference:held').length, 2, 'no release on what the released start stored'); + assert.equal(backlog.compiles, compiles); + } finally { + await t.close(); + } + }); + }); + + it('does not release a re-refused turn again on the clock row its own start stored', async () => { + await withBacklog({ alwaysPending: true, stalled: true }, async (backlog) => { + const t = await residentTurn({ maintenanceIntervalMs: 50, clock: { intervalSeconds: 1 } }); + try { + t.send('Are you there?'); + assert.ok(await pollUntil(() => t.of('inference:held').length === 1 && backlog.previews >= 1, 5_000)); + const cm = t.framework.getAgent('prime')!.getContextManager(); + const clockRows = (): number => (cm.queryMessages({}).messages as unknown as Array<{ metadata?: { kind?: string } }>) + .filter((m) => m.metadata?.kind === 'clock').length; + const before = clockRows(); + await new Promise((r) => setTimeout(r, 1_100)); + backlog.fit = true; + backlog.refuseCompiles = Infinity; + const messages = cm.queryMessages({}).messages; + cm.editMessage(messages[messages.length - 1]!.id, [{ type: 'text', text: 'Hi?' }]); + assert.ok(await pollUntil(() => t.of('inference:held').length === 2, 5_000), 'released on the edit, refused, and held again'); + assert.equal(clockRows(), before + 1, 'the released start stored a clock row, a boundary after the last'); + const compiles = backlog.compiles; + await new Promise((r) => setTimeout(r, 600)); + assert.equal(t.of('inference:held').length, 2, 'no release on the clock row the released start stored'); + assert.equal(backlog.compiles, compiles); + } finally { + await t.close(); + } + }); + }); + + it('looks again when the budget its next compile plans at changes, as when a tool-result reserve settles', async () => { + await withBacklog({ stalled: true }, async (backlog) => { + const t = await residentTurn({ maintenanceIntervalMs: 50 }); + let reserve = 5_000; + const agent = t.framework.getAgent('prime') as unknown as { toolResultGuard: object }; + Object.defineProperty(agent.toolResultGuard, 'pendingWireReserveTokens', { get: () => reserve, configurable: true }); + try { + t.send('Are you there?'); + assert.ok(await pollUntil(() => t.of('inference:held').length === 1 && backlog.previews >= 1, 5_000)); + await new Promise((r) => setTimeout(r, 200)); + const previews = backlog.previews; + const before = backlog.previewBudgets.at(-1) as { reserveForResponse: number }; + reserve = 0; + assert.ok(await pollUntil(() => backlog.previews > previews, 2_000), 'a smaller reserve, which can make the context fit, brings a look'); + assert.equal((backlog.previewBudgets.at(-1) as { reserveForResponse: number }).reserveForResponse, before.reserveForResponse - 5_000, + 'at the budget the next compile plans at, with the reserve settled'); + } finally { + await t.close(); + } + }); + }); + + it('for a strategy that reports no progress, does not release a re-refused turn again until compression finishes', async () => { + await withBacklog({ fit: true, alwaysPending: true, refuseCompiles: Infinity, noProgress: true }, async (backlog) => { + const t = await residentTurn({ maintenanceIntervalMs: 50 }); + try { + t.send('Are you there?'); + assert.ok(await pollUntil(() => t.of('inference:held').length === 2, 5_000), 'released once, refused, and held again'); + const compiles = backlog.compiles; + const asked = t.looks(); + await new Promise((r) => setTimeout(r, 600)); + assert.ok(t.looks() > asked, 'passes asked the hold to look'); + assert.equal(t.of('inference:held').length, 2, 'no release pass after pass, with no progress to go on'); + assert.equal(backlog.compiles, compiles); + backlog.alwaysPending = false; + assert.ok(await pollUntil(() => t.of('inference:hold_ended').length === 2, 3_000), 'it looks again once nothing is pending'); + } finally { + await t.close(); + } + }); + }); + + it('looks again when compression moved while a released turn compiled, though that compile was refused', async () => { + await withBacklog({ fit: true, alwaysPending: true, refuseCompiles: 2, progress: 0, slowCompile: { nth: 2, ms: 300 } }, async (backlog) => { + const t = await residentTurn({ maintenanceIntervalMs: 50 }); + try { + t.send('Are you there?'); + assert.ok(await pollUntil(() => backlog.compiles === 2, 5_000), 'released, and its compile has begun'); + backlog.progress = 1; + assert.ok(await pollUntil(() => storedText(t.framework).includes('Answered once it fit'), 5_000), + 'the step that landed during the refused compile brought a look, which released the turn again'); + assert.equal(t.of('inference:held').length, 2); + } finally { + await t.close(); + } + }); + }); + + it('looks again when a message is removed while held', async () => { + await withBacklog({ stalled: true }, async (backlog) => { + const t = await residentTurn({ maintenanceIntervalMs: 50 }); + try { + t.send('Are you there?'); + assert.ok(await pollUntil(() => t.of('inference:held').length === 1 && backlog.previews >= 1, 5_000)); + await new Promise((r) => setTimeout(r, 200)); + const previews = backlog.previews; + const asked = t.looks(); + await new Promise((r) => setTimeout(r, 300)); + assert.ok(t.looks() > asked, 'passes asked the hold to look'); + assert.equal(backlog.previews, previews, 'and nothing had changed, so it took no verdict'); + const cm = t.framework.getAgent('prime')!.getContextManager(); + const messages = cm.queryMessages({}).messages; + cm.removeMessage(messages[messages.length - 1]!.id); + assert.ok(await pollUntil(() => backlog.previews > previews, 2_000), 'a removed message brings a look'); + } finally { + await t.close(); + } + }); + }); + + it('looks again when a message is removed and another arrives between two looks', async () => { + await withBacklog({ stalled: true }, async (backlog) => { + const t = await residentTurn({ maintenanceIntervalMs: 50 }); + try { + t.send('Are you there?'); + assert.ok(await pollUntil(() => t.of('inference:held').length === 1 && backlog.previews >= 1, 5_000)); + await new Promise((r) => setTimeout(r, 200)); + const previews = backlog.previews; + const cm = t.framework.getAgent('prime')!.getContextManager(); + const length = cm.getMessageCount(); + const messages = cm.queryMessages({}).messages; + cm.removeMessage(messages[messages.length - 1]!.id); + cm.addMessage('User', [{ type: 'text', text: 'Still there?' }]); + assert.equal(cm.getMessageCount(), length, 'the store is as long as it was'); + assert.ok(await pollUntil(() => backlog.previews > previews, 2_000), 'yet the removal brings a look'); + } finally { + await t.close(); + } + }); + }); + + it('looks again when a message is edited while held', async () => { + await withBacklog({ stalled: true }, async (backlog) => { + const t = await residentTurn({ maintenanceIntervalMs: 50 }); + try { + t.send('Are you there?'); + assert.ok(await pollUntil(() => t.of('inference:held').length === 1 && backlog.previews >= 1, 5_000)); + await new Promise((r) => setTimeout(r, 200)); + const previews = backlog.previews; + const cm = t.framework.getAgent('prime')!.getContextManager(); + const messages = cm.queryMessages({}).messages; + cm.editMessage(messages[messages.length - 1]!.id, [{ type: 'text', text: 'Hi?' }]); + assert.ok(await pollUntil(() => backlog.previews > previews, 2_000), 'an edit, which can shrink the context, brings a look'); + } finally { + await t.close(); + } + }); + }); + + it('ends a released turn refused with nothing pending as a hold with nothing left to do', async () => { + await withBacklog({ fit: true, alwaysPending: true, refuseCompiles: Infinity, slowCompile: { nth: 2, ms: 300 } }, async (backlog) => { + const t = await residentTurn({ maintenanceIntervalMs: 50 }); + try { + t.send('Are you there?'); + assert.ok(await pollUntil(() => backlog.compiles === 2, 5_000), 'released, and its compile has begun'); + backlog.alwaysPending = false; + assert.ok(await pollUntil(() => t.of('inference:exhausted').length === 1, 5_000)); + assert.equal(t.of('inference:held').length, 1, 'not held again'); + assert.match(String(t.of('inference:exhausted')[0]!.error), + /the turn was first held \d+ s ago; compression had nothing left to do and it still didn't fit/); + } finally { + await t.close(); + } + }); + }); + + it('starts no hold for a compile refused once stop() has closed admission', async () => { + await withBacklog({ slowCompile: { nth: 1, ms: 300 } }, async (backlog) => { + const t = await residentTurn({ maintenanceIntervalMs: 50 }); + t.send('Are you there?'); + assert.ok(await pollUntil(() => backlog.compiles === 1, 5_000), 'the turn\'s compile has begun'); + await t.close(); + assert.ok(await pollUntil(() => t.of('inference:failed').length === 1, 3_000), 'the compile was refused'); + assert.deepEqual(t.of('inference:held'), [], 'and no hold began, since stop had closed admission'); + }); + }); + + it('holds a context refusal as it holds an over-budget one', async () => { + await withBacklog({ fitAt: 40, refusalName: 'UncoveredDropError' }, async () => { + const t = await residentTurn({ maintenanceIntervalMs: 50 }); + try { + t.send('Are you there?'); + assert.ok(await pollUntil(() => storedText(t.framework).includes('Answered once it fit'), 10_000)); + const held = t.of('inference:held'); + assert.equal(held.length, 1); + assert.equal(held[0]!.errorType, 'context_refusal'); + assert.deepEqual(t.of('inference:hold_ended').map((e) => e.outcome), ['fit']); + assert.deepEqual(t.of('inference:exhausted'), []); + } finally { + await t.close(); + } + }); + }); + + it('fails a released turn refused after its hold\'s deadline, rather than holding it afresh', async () => { + await withBacklog({ fit: true, alwaysPending: true, refuseCompiles: Infinity, slowCompile: { nth: 2, ms: 500 } }, async () => { + const t = await residentTurn({ maintenanceIntervalMs: 50, holdMs: 300 }); + try { + t.send('Are you there?'); + assert.ok(await pollUntil(() => t.of('inference:hold_ended').length === 1, 3_000)); + assert.equal(heldRequests(t.framework), undefined, 'a released turn is not waiting, so the status shows no hold'); + assert.ok(await pollUntil(() => t.of('inference:exhausted').length === 1, 5_000)); + assert.equal(t.of('inference:held').length, 1, 'no second hold past the first one\'s deadline'); + assert.match(String(t.of('inference:exhausted')[0]!.error), /the hold's 1 s bound passed and it still didn't fit/); + } finally { + await t.close(); + } + }); + }); + + it('says when a spent turn was first held as of each refusal it reports', async () => { + await withBacklog({ slowCompile: { nth: 2, ms: 500 } }, async () => { + const t = await residentTurn({ maintenanceIntervalMs: 3_600_000, holdMs: 200 }); + try { + t.send('Are you there?'); + assert.ok(await pollUntil(() => t.of('inference:hold_ended').length === 1, 3_000)); + assert.deepEqual(t.of('inference:hold_ended').map((e) => e.outcome), ['deadline']); + (t.internal.budgetHolds as unknown as Map).get('prime')!.startedAt -= 600_000; + assert.ok(await pollUntil(() => t.of('inference:exhausted').length === 1, 5_000)); + assert.match(String(t.of('inference:exhausted')[0]!.error), /the turn was first held 10 min ago; the hold's/); + } finally { + await t.close(); + } + }); + }); + + it('says a wait of minutes in minutes, in the note and in the failure', async () => { + type Times = { state: string; startedAt: number; until: number }; + const times = (t: { internal: Internal }) => (t.internal.budgetHolds as unknown as Map).get('prime')!; + await withBacklog({}, async (backlog) => { + const t = await residentTurn({ maintenanceIntervalMs: 50 }); + try { + t.send('Are you there?'); + assert.ok(await pollUntil(() => t.of('inference:held').length === 1, 5_000)); + times(t).startedAt = Date.now() - 7 * 60_000; + backlog.fit = true; + assert.ok(await pollUntil(() => t.membrane.calls.length === 1, 5_000)); + assert.ok(JSON.stringify(t.membrane.calls[0]).includes( + "[budget-hold] Your context didn't fit its budget (100,201 tokens against 48,568), so this turn waited 7 min for compression to make room.")); + } finally { + await t.close(); + } + }); + await withBacklog({ fit: true, alwaysPending: true, refuseCompiles: Infinity, slowCompile: { nth: 2, ms: 300 } }, async (backlog) => { + const t = await residentTurn({ maintenanceIntervalMs: 50 }); + try { + t.send('Are you there?'); + assert.ok(await pollUntil(() => times(t)?.state === 'released' && backlog.compiles === 2, 5_000), 'released, and its compile has begun'); + times(t).startedAt = Date.now() - 30 * 60_000; + times(t).until = Date.now() - 1; + assert.ok(await pollUntil(() => t.of('inference:exhausted').length === 1, 5_000)); + assert.ok(String(t.of('inference:exhausted')[0]!.error).endsWith( + "(the turn was first held 30 min ago; the hold's 30 min bound passed and it still didn't fit)"), + String(t.of('inference:exhausted')[0]!.error)); + } finally { + await t.close(); + } + }); + }); + + it('ends a released turn the inference policy declines: nothing of its hold reaches a later turn', async () => { + await withBacklog({}, async (backlog) => { + let decline = false; + const t = await residentTurn({ maintenanceIntervalMs: 50, inferencePolicy: { shouldInfer: () => !decline } }); + try { + t.send('Are you there?'); + assert.ok(await pollUntil(() => t.of('inference:held').length === 1, 5_000)); + const openLogical = (t.framework as unknown as { openLogical: Map }).openLogical; + assert.ok(openLogical.has('prime'), 'the held turn\'s logical turn stays open'); + decline = true; + backlog.fit = true; + assert.ok(await pollUntil(() => t.of('inference:hold_ended').length === 1 && t.internal.budgetHolds.size === 0, 3_000), + 'released, declined, and its record gone'); + assert.equal(t.membrane.calls.length, 0); + assert.ok(!openLogical.has('prime'), 'and the decline closed it'); + decline = false; + t.send('A later turn'); + assert.ok(await pollUntil(() => t.membrane.calls.length === 1, 3_000)); + assert.ok(!JSON.stringify(t.membrane.calls[0]).includes('[budget-hold]'), 'the later turn carries no note of the declined one\'s wait'); + } finally { + await t.close(); + } + }); + }); + + it('ends a released turn that a stop reaches before it sends anything: its record goes, and a later turn gets no note of its own', async () => { + let open!: () => void; + const until = new Promise((resolve) => { open = resolve; }); + await withBacklog({ compileGate: { nth: 2, until } }, async (backlog) => { + const t = await residentTurn({ maintenanceIntervalMs: 50, responses: 2 }); + try { + t.send('Are you there?'); + assert.ok(await pollUntil(() => t.of('inference:held').length === 1, 5_000)); + backlog.fit = true; + assert.ok(await pollUntil(() => backlog.compiles === 2, 3_000), 'the hold released the turn into its compile'); + assert.equal(t.framework.abortInference('prime'), true, 'a stop made while the released turn compiles is reported taken'); + open(); + assert.ok(await pollUntil(() => t.of('inference:aborted').length === 1, 3_000), 'the stop ended the turn at the check after its compile'); + assert.equal(t.of('inference:completed').length, 0, 'the stopped turn completed nothing'); + assert.equal(t.internal.budgetHolds.size, 0, 'the hold\'s record went with the turn'); + t.send('A later turn'); + assert.ok(await pollUntil(() => t.of('inference:completed').length === 1, 3_000), 'the later turn ran'); + const later = t.membrane.calls[t.membrane.calls.length - 1]; + assert.equal(JSON.stringify(later).split('[budget-hold]').length - 1, 1, + 'the later turn reads the note the stopped turn\'s start stored, and gets none of its own'); + } finally { + open(); + await t.close(); + } + }); + }); + + it('ends a turn whose compile a stop reached, rather than holding it, when that compile is refused for budget', async () => { + let open!: () => void; + const until = new Promise((resolve) => { open = resolve; }); + await withBacklog({ compileGate: { nth: 1, until } }, async (backlog) => { + const t = await residentTurn({ maintenanceIntervalMs: 50 }); + try { + t.send('Are you there?'); + assert.ok(await pollUntil(() => backlog.compiles === 1, 5_000), 'the turn is in its compile'); + assert.equal(t.framework.abortInference('prime'), true, 'a stop made while the turn compiles is reported taken'); + open(); + assert.ok(await pollUntil(() => t.of('inference:aborted').length === 1, 3_000), 'the stop ended the turn at its failure'); + assert.equal(t.of('inference:held').length, 0, 'the refused turn was not held'); + assert.equal(t.internal.budgetHolds.size, 0); + assert.equal(t.membrane.calls.length, 0, 'nothing was sent'); + } finally { + open(); + await t.close(); + } + }); + }); + + it('holds a conversation agent\'s refused turn as it holds a resident\'s', async () => { + await withBacklog({ fit: true }, async (backlog) => { + const t = await residentTurn({ maintenanceIntervalMs: 50 }); + const fw = t.framework as unknown as { + conversationRouter: unknown; + createConversationAgent: (name: string, channelId: string) => Promise; + }; + const fork = 'prime--chanA'; + try { + fw.conversationRouter = { + templateAgent: 'prime', expired: () => [], channelForAgent: () => 'chanA', touch: () => {}, + getBinding: () => undefined, closurePrompt: () => undefined, + }; + await fw.createConversationAgent(fork, 'chanA'); + backlog.fit = false; + backlog.fitAt = backlog.ticks + 12; + t.internal.pendingRequests.push({ agentName: fork, reason: 'external-message', source: 'test', timestamp: Date.now() } as never); + assert.ok(await pollUntil(() => t.membrane.calls.length === 1, 10_000), 'the fork\'s turn runs once its context fits'); + const held = t.of('inference:held'); + assert.deepEqual(held.map((e) => e.agentName), [fork], 'the fork\'s refused turn was held'); + assert.deepEqual(t.of('inference:hold_ended').map((e) => [e.agentName, e.outcome]), [[fork, 'fit']]); + assert.deepEqual(t.of('inference:exhausted'), []); + } finally { + fw.conversationRouter = undefined; + await t.close(); + } + }); + }); + + it('leaves nothing of a hold after a restart: its messages stay, and the next wake compiles afresh', async () => { + const dir = mkdtempSync(join(tmpdir(), 'budget-hold-restart-')); + try { + await withBacklog({}, async (backlog) => { + const first = await residentTurn({ maintenanceIntervalMs: 50, storePath: join(dir, 'store') }); + try { + first.send('Are you there?'); + assert.ok(await pollUntil(() => first.of('inference:held').length === 1, 5_000)); + } finally { + await first.close(); + } + assert.deepEqual(first.of('inference:hold_ended').map((e) => e.outcome), ['stopped']); + backlog.fit = true; + const second = await residentTurn({ maintenanceIntervalMs: 50, storePath: join(dir, 'store') }); + try { + assert.equal(heldRequests(second.framework), undefined, 'no hold after the restart'); + assert.ok(storedText(second.framework).includes('Are you there?'), 'the held turn\'s message is in the store'); + second.send('Still there?'); + assert.ok(await pollUntil(() => second.membrane.calls.length === 1, 5_000)); + const request = JSON.stringify(second.membrane.calls[0]); + assert.ok(request.includes('Are you there?') && request.includes('Still there?'), + 'the next wake compiles afresh, from the store'); + assert.deepEqual(second.of('inference:held'), []); + } finally { + await second.close(); + } + }); + } finally { + rmSync(dir, { recursive: true, force: true }); + } + }); + + it('leaves an ephemeral run to fail as before: its caller is waiting', async () => { + await withBacklog({}, async () => { + const tempDir = mkdtempSync(join(tmpdir(), 'budget-hold-eph-')); + const framework = await AgentFramework.create({ + storePath: join(tempDir, 'store'), + membrane: new MockMembrane().asMembrane(), + agents: [], + modules: [], + syncIntervalMs: 0, + maintenanceIntervalMs: 50, + errorPolicy: { maxRetries: 3, onInferenceError: (_error, _agent, attempt) => (attempt < 3 ? { retry: true, delayMs: 0 } : { retry: false }) }, + }); + const traces: TraceEvent[] = []; + framework.onTrace((e) => traces.push(e)); + try { + const { agent, contextManager } = await framework.createEphemeralAgent({ + name: 'worker', model: 'test-model', systemPrompt: 'Do the task.', allowedTools: 'all', + }); + contextManager.addMessage('user', [{ type: 'text', text: 'Run once.' }]); + const run = framework.runEphemeralToCompletion(agent, contextManager); + framework.start(); + let timer: ReturnType | undefined; + const bound = new Promise((_resolve, reject) => { + timer = setTimeout(() => reject(new Error('the run was still waiting after 5 s')), 5_000); + }); + try { + await assert.rejects(Promise.race([run, bound]), /exceed hard budget/); + } finally { + clearTimeout(timer); + } + assert.deepEqual(traces.filter((e) => e.type === 'inference:held'), []); + } finally { + await framework.stop(); + rmSync(tempDir, { recursive: true, force: true }); + } + }); + }); + + it('holds a batch-only start\'s request as an ordinary cause, which no cancellation withdraws', async () => { + await withBacklog({}, async (backlog) => { + const t = await residentTurn({ maintenanceIntervalMs: 50 }); + try { + t.internal.pendingRequests.push({ + agentName: 'prime', reason: 'mcpl:push-event', source: 'test', timestamp: Date.now(), + coalescingSubject: 'zz-subject', coalescingBatch: true, coalescingBatchSubjects: ['zz-subject'], + } as unknown as { agentName: string }); + assert.ok(await pollUntil(() => t.of('inference:held').length === 1, 5_000)); + const held = holdOf(t)!.heldRequests; + assert.equal(held.length, 1); + assert.equal(held[0]!.coalescingBatch, undefined, 'no longer batch-only, so the released start keeps its turn'); + assert.equal(held[0]!.coalescingBatchSubjects, undefined); + (t.framework as unknown as { cancelCoalescedWakes: (subject: string) => void }).cancelCoalescedWakes('zz-subject'); + assert.equal(holdOf(t)!.heldRequests.length, 1, 'the turn keeps its own cause'); + backlog.fit = true; + assert.ok(await pollUntil(() => storedText(t.framework).includes('Answered once it fit'), 3_000)); + } finally { + await t.close(); + } + }); + }); + + it('lets a withdrawn coalesced wake leave a hold, as it leaves a provider hold', async () => { + await withBacklog({}, async () => { + const t = await residentTurn({ maintenanceIntervalMs: 50 }); + try { + t.send('Are you there?'); + assert.ok(await pollUntil(() => t.of('inference:held').length === 1, 5_000)); + t.internal.pendingRequests.push({ + agentName: 'prime', reason: 'mcpl:push-event', source: 'test', timestamp: Date.now(), coalescingSubject: 'zz-retracted', + } as unknown as { agentName: string }); + assert.ok(await pollUntil(() => holdOf(t)?.heldRequests.length === 2, 5_000), 'the wake joins the hold'); + (t.framework as unknown as { cancelCoalescedWakes: (subject: string) => void }).cancelCoalescedWakes('zz-retracted'); + assert.deepEqual(holdOf(t)!.heldRequests.map((r) => r.coalescingSubject), [undefined], 'the withdrawn wake is gone, the turn\'s own stays'); + } finally { + await t.close(); + } + }); + }); + + it('keeps a closing conversation\'s held closure turn from the reaper', async () => { + await withBacklog({}, async () => { + const t = await residentTurn({ maintenanceIntervalMs: 50 }); + const fw = t.framework as unknown as { + conversationRouter: unknown; lastConversationSweep: number; closingConversationAgents: Set; + sweepExpiredConversations: () => void; agents: Map; + }; + try { + t.send('Are you there?'); + assert.ok(await pollUntil(() => t.of('inference:held').length === 1, 5_000)); + fw.conversationRouter = { expired: () => [] }; + fw.lastConversationSweep = 0; + fw.closingConversationAgents.add('prime'); + fw.sweepExpiredConversations(); + assert.ok(fw.agents.has('prime'), 'the held closure turn is unfinished work'); + assert.ok(holdOf(t), 'and its hold stays'); + } finally { + fw.closingConversationAgents.delete('prime'); + fw.conversationRouter = undefined; + await t.close(); + } + }); + }); + + it('drops the hold\'s record when the released start fails outside its compile', async () => { + await withBacklog({}, async (backlog) => { + const t = await residentTurn({ maintenanceIntervalMs: 50 }); + const fw = t.framework as unknown as { recordTurnCheckpoint: (name: string) => void }; + try { + t.send('Are you there?'); + assert.ok(await pollUntil(() => t.of('inference:held').length === 1, 5_000)); + assert.ok(await pollUntil(() => !t.internal.activeTurnTokens.has('prime'), 3_000)); + const original = fw.recordTurnCheckpoint; + let failed = 0; + fw.recordTurnCheckpoint = function (this: unknown, name: string) { + if (failed++ === 0) throw new Error('ENOSPC: no space left on device'); + return original.call(this, name); + }; + backlog.fit = true; + assert.ok(await pollUntil(() => failed > 0 && t.internal.budgetHolds.size === 0, 3_000), + 'nothing of the failed turn\'s hold is left for the next'); + } finally { + await t.close(); + } + }); + }); + + it('opens the event gate again when a held continuation resumes', async () => { + const proto = ContextManager.prototype as unknown as Patchable; + const original = { compile: proto.compile, tick: proto.tick, isReady: proto.isReady }; + let real = 0; + proto.compile = async function (this: unknown, ...args: unknown[]) { + const dryRun = (args[2] as { dryRun?: boolean } | undefined)?.dryRun === true; + if (!dryRun && ++real === 2) { + throw Object.assign(new Error('Compile plan would exceed hard budget: budget=48568'), { name: 'OverBudgetError', budget: 48_568, actual: 49_000 }); + } + return original.compile.apply(this, args); + }; + proto.tick = async function () {}; + proto.isReady = function () { return false; }; + const tempDir = mkdtempSync(join(tmpdir(), 'budget-hold-gate-')); + const membrane = new SequentialStreamMembrane(); + membrane.pushResponse(nearCapToolResponse()); + membrane.pushResponse(createMockResponse([{ type: 'text', text: 'Answered after the restart' }])); + const framework = await AgentFramework.create({ + storePath: join(tempDir, 'store'), + membrane: membrane.asMembrane(), + agents: [{ name: 'prime', model: 'test-model', systemPrompt: 'You are prime.', allowedTools: 'all', maxTokens: 4_000, physicalWindowTokens: 200_000 }], + modules: [new BlobToolModule()], + syncIntervalMs: 0, + maintenanceIntervalMs: 50, + gate: { config: { policies: [], default: 'skip' } }, + errorPolicy: { maxRetries: 3, onInferenceError: (_error: Error, _agent: string, attempt: number) => (attempt < 3 ? { retry: true, delayMs: 0 } : { retry: false }) }, + }); + const gate = (framework as unknown as { eventGate: { inferenceDiagnostics(): { inferring: string[] } } }).eventGate; + const traces: TraceEvent[] = []; + const inferringDuringResume: string[][] = []; + framework.onTrace((e) => { + traces.push(e); + if (e.type === 'inference:content_block' && traces.some((x) => x.type === 'inference:hold_ended')) { + inferringDuringResume.push(gate.inferenceDiagnostics().inferring); + } + }); + try { + framework.start(); + framework.getAgent('prime')!.getContextManager().addMessage('user', [{ type: 'text', text: 'fetch it' }]); + (framework as unknown as { pendingRequests: unknown[] }).pendingRequests.push({ agentName: 'prime', reason: 'external-message', source: 'test', timestamp: Date.now() }); + assert.ok(await pollUntil(() => storedText(framework).includes('Answered after the restart'), 10_000)); + const held = traces.filter((e) => e.type === 'inference:held'); + assert.equal(held.length, 1, 'the restart\'s refused compile was held'); + assert.ok(inferringDuringResume.length >= 1); + assert.ok(inferringDuringResume.every((names) => names.includes('prime')), 'the resumed turn is inferring at the gate, so its events buffer'); + assert.ok(await pollUntil(() => !gate.inferenceDiagnostics().inferring.includes('prime'), 2_000), 'the gate closes when the turn ends'); + } finally { + await framework.stop(); + Object.assign(proto, original); + rmSync(tempDir, { recursive: true, force: true }); + } + }); + + it('ends the held turn and drops the hold, its requests and what it left for the name, at stop and at disposal', async () => { + await withBacklog({}, async () => { + const t = await residentTurn({ maintenanceIntervalMs: 50 }); + let closed = false; + try { + t.send('Are you there?'); + assert.ok(await pollUntil(() => t.of('inference:held').length === 1, 5_000)); + await t.close(); + closed = true; + assert.deepEqual(t.of('inference:hold_ended').map((e) => e.outcome), ['stopped']); + assert.equal(t.of('inference:aborted').length, 1, 'the stop ended the held turn'); + assert.deepEqual(t.internal.pendingRequests.filter((r) => r.agentName === 'prime'), [], 'the held request went with the stop'); + assert.equal(t.membrane.calls.length, 0); + assert.equal(t.internal.budgetHolds.size, 0); + } finally { + if (!closed) await t.close(); + } + }); + await withBacklog({}, async () => { + const t = await residentTurn({ maintenanceIntervalMs: 50, holdMs: 300 }); + try { + t.send('Are you there?'); + assert.ok(await pollUntil(() => t.of('inference:held').length === 1, 5_000)); + t.internal.disposeConversationAgent('prime'); + assert.deepEqual(t.of('inference:hold_ended').map((e) => e.outcome), ['stopped']); + assert.equal(t.of('inference:aborted').length, 1, 'the disposal ended the held turn'); + assert.equal(t.internal.budgetHolds.size, 0, 'a later agent of the name starts with none of it'); + await new Promise((r) => setTimeout(r, 450)); + assert.deepEqual(t.internal.pendingRequests.filter((r) => r.agentName === 'prime'), []); + assert.deepEqual(t.of('inference:hold_ended').map((e) => e.outcome), ['stopped']); + } finally { + await t.close(); + } + }); + }); + it('lets a quiesce wait for a held continuation, which runs once its hold ends while the wakes that joined it wait for resume', async () => { + const t = await heldRestart(['Answered after the restart', 'Answered the wake']); + try { + assert.ok(await pollUntil(() => t.of('inference:held').length === 1, 10_000), 'the restart is held'); + t.internal.pendingRequests.push({ agentName: 'prime', reason: 'external-message', source: 'test', timestamp: Date.now() } as unknown as { agentName: string }); + assert.ok(await pollUntil(() => holdOf(t)?.heldRequests.length === 2, 3_000), 'a wake joins the hold'); + const status = await t.framework.quiesce({ reason: 'test', timeoutMs: 1_000 }); + assert.equal(status.drained, false, 'the held turn is alive, so the drain waits for it'); + assert.deepEqual(status.activeTurnAgents, ['prime']); + assert.equal(t.membrane.calls.length, 1); + t.fit(); + assert.ok(await pollUntil(() => storedText(t.framework).includes('Answered after the restart'), 5_000), + 'the continuation runs while the host is quiesced, as a turn alive at the quiesce does'); + assert.ok(await pollUntil(() => t.framework.getHostModeStatus().drained, 3_000), 'and the host is then drained'); + assert.equal(t.framework.getHostModeStatus().gatedRequests, 1, 'with the joined wake waiting'); + assert.equal(t.membrane.calls.length, 2); + await t.framework.resume(); + assert.ok(await pollUntil(() => storedText(t.framework).includes('Answered the wake'), 5_000), 'the wake runs at resume'); + assert.equal(t.membrane.calls.length, 3); + } finally { + await t.close(); + } + }); + + it('ends a held turn at a quiesce\'s abandon, and the wakes that joined it wait for resume', async () => { + await withBacklog({}, async (backlog) => { + const t = await residentTurn({ maintenanceIntervalMs: 50 }); + try { + t.send('Are you there?'); + assert.ok(await pollUntil(() => t.of('inference:held').length === 1, 5_000)); + t.send('Joined while held'); + assert.ok(await pollUntil(() => heldRequests(t.framework) === 2, 5_000)); + const status = await t.framework.quiesce({ reason: 'test', timeoutMs: 1_000, abandon: true }); + assert.equal(status.drained, true, 'the abandon ended the held turn'); + assert.deepEqual(t.of('inference:hold_ended').map((e) => e.outcome), ['stopped']); + assert.equal(t.of('inference:aborted').length, 1, 'one terminal event for the turn'); + assert.equal(t.internal.budgetHolds.size, 0); + assert.equal(status.gatedRequests, 1, 'the wake that joined it waits for resume'); + await t.framework.switchBranch('main'); + backlog.fit = true; + await t.framework.resume(); + assert.ok(await pollUntil(() => storedText(t.framework).includes('Answered once it fit'), 5_000), + 'the joined wake runs as a turn of its own at resume'); + assert.equal(t.membrane.calls.length, 1); + } finally { + await t.close(); + } + }); + }); + + it('ends a held turn at a caller\'s stop, and the wakes that joined it run as a turn of their own', async () => { + await withBacklog({}, async (backlog) => { + const t = await residentTurn({ maintenanceIntervalMs: 3_600_000 }); + try { + t.send('Are you there?'); + assert.ok(await pollUntil(() => t.of('inference:held').length === 1 && backlog.previews >= 1, 5_000)); + t.send('Joined while held'); + assert.ok(await pollUntil(() => heldRequests(t.framework) === 2, 5_000)); + backlog.fit = true; + assert.equal(t.framework.abortInference('prime'), true, 'the stop is reported taken'); + assert.deepEqual(t.of('inference:hold_ended').map((e) => e.outcome), ['stopped']); + assert.equal(t.of('inference:aborted').length, 1, 'one terminal event for the turn'); + assert.ok(await pollUntil(() => storedText(t.framework).includes('Answered once it fit'), 5_000), + 'the joined wake runs as a turn of its own'); + assert.equal(t.membrane.calls.length, 1); + assert.equal(t.of('inference:held').length, 1); + } finally { + await t.close(); + } + }); + }); + + it('reports a stopped hold\'s end once its turn has ended and the wakes that joined it are back in the queue', async () => { + await withBacklog({}, async (backlog) => { + const t = await residentTurn({ maintenanceIntervalMs: 3_600_000 }); + try { + t.send('Are you there?'); + assert.ok(await pollUntil(() => t.of('inference:held').length === 1 && backlog.previews >= 1, 5_000)); + t.send('Joined while held'); + assert.ok(await pollUntil(() => heldRequests(t.framework) === 2, 5_000)); + let atEnd: { aborted: number; queued: number } | undefined; + t.framework.onTrace((e) => { + if (e.type === 'inference:hold_ended') { + atEnd = { aborted: t.of('inference:aborted').length, queued: t.internal.pendingRequests.filter((r) => r.agentName === 'prime').length }; + } + }); + assert.equal(t.framework.abortInference('prime'), true); + assert.deepEqual(atEnd, { aborted: 1, queued: 1 }, 'the turn had ended and its joined wake was queued when the end was reported'); + } finally { + await t.close(); + } + }); + }); + + it('reports a stopped hold\'s end even when the turn\'s teardown throws', async () => { + await withBacklog({}, async (backlog) => { + const t = await residentTurn({ maintenanceIntervalMs: 3_600_000 }); + const fw = t.framework as unknown as { endContinuedTurn: (...args: unknown[]) => void }; + try { + t.send('Are you there?'); + assert.ok(await pollUntil(() => t.of('inference:held').length === 1 && backlog.previews >= 1, 5_000)); + fw.endContinuedTurn = () => { throw new Error('teardown broke'); }; + assert.throws(() => t.framework.abortInference('prime'), /teardown broke/); + assert.deepEqual(t.of('inference:hold_ended').map((e) => e.outcome), ['stopped']); + } finally { + delete (fw as Partial).endContinuedTurn; + await t.close(); + } + }); + }); + + it('gives a deferred batch\'s wake that joins a held continuation a fresh turn of its own', async () => { + const t = await heldRestart(['Answered after the restart', 'Answered the batch']); + try { + assert.ok(await pollUntil(() => t.of('inference:held').length === 1, 10_000), 'the restart is held'); + t.internal.pendingRequests.push({ + agentName: 'prime', reason: 'mcpl:push-event', source: 'test', timestamp: Date.now(), + coalescingSubject: 'zz-subject', coalescingBatch: true, coalescingBatchSubjects: ['zz-subject'], + } as unknown as { agentName: string }); + assert.ok(await pollUntil(() => holdOf(t)?.heldRequests.length === 2, 3_000), 'the batch\'s wake joins the hold'); + t.fit(); + assert.ok(await pollUntil(() => storedText(t.framework).includes('Answered after the restart'), 5_000), 'the continuation runs'); + assert.ok(await pollUntil(() => storedText(t.framework).includes('Answered the batch'), 5_000), 'the batch\'s wake gets a fresh turn'); + assert.equal(t.membrane.calls.length, 3); + } finally { + await t.close(); + } + }); + + it('holds nothing in a framework that was never started: the refusal takes the error policy\'s path, as before', async () => { + await withBacklog({}, async () => { + const tempDir = mkdtempSync(join(tmpdir(), 'budget-hold-unstarted-')); + const framework = await AgentFramework.create({ + storePath: join(tempDir, 'store'), + membrane: new SequentialStreamMembrane().asMembrane(), + agents: [{ name: 'prime', model: 'test-model', systemPrompt: 'You are prime.' }], + modules: [new InboxModule()], + syncIntervalMs: 0, + maintenanceIntervalMs: 50, + errorPolicy: { maxRetries: 3, onInferenceError: (_error: Error, _agent: string, attempt: number) => (attempt < 3 ? { retry: true, delayMs: 0 } : { retry: false }) }, + }); + const traces: TraceEvent[] = []; + framework.onTrace((e) => traces.push(e)); + try { + framework.pushEvent({ + type: 'external-message', source: 'test', content: [{ type: 'text', text: 'Are you there?' }], metadata: {}, triggerInference: true, + } as unknown as ProcessEvent); + await framework.runUntilIdle(); + assert.equal(traces.filter((e) => e.type === 'inference:held').length, 0, 'no run loop would run a released turn'); + const exhausted = traces.filter((e) => e.type === 'inference:exhausted') as unknown as Array<{ errorType?: string }>; + assert.equal(exhausted.length, 1, 'the turn fails before runUntilIdle returns'); + assert.equal(exhausted[0]!.errorType, 'over_budget', 'from the refusal, by the error policy\'s path'); + } finally { + await framework.stop(); + rmSync(tempDir, { recursive: true, force: true }); + } + }); + }); + + it('looks again when compression stops being pending though the strategy\'s progress snapshot stands still', async () => { + await withBacklog({ progress: 1, readyAt: 30 }, async (backlog) => { + const t = await residentTurn({ maintenanceIntervalMs: 50 }); + try { + t.send('Are you there?'); + assert.ok(await pollUntil(() => t.of('inference:held').length === 1, 5_000)); + assert.ok(await pollUntil(() => backlog.ticks >= 30, 5_000), 'compression stops being pending'); + assert.ok(await pollUntil(() => t.of('inference:hold_ended').length === 1, 3_000), 'the hold ends without waiting for its deadline'); + assert.equal(t.of('inference:hold_ended')[0]!.outcome, 'nothing_pending'); + } finally { + await t.close(); + } + }); + }); + + it('keeps the undo history at its limit as a turn without a hold leaves it', async () => { + await withBacklog({}, async (backlog) => { + const t = await residentTurn({ maintenanceIntervalMs: 50 }); + const checkpoints = t.framework as unknown as { + getTurnCheckpoints: (agentName: string) => Array<{ turnIndex: number }>; + saveTurnCheckpoints: (agentName: string, checkpoints: unknown[]) => void; + turnCounters: Map; + }; + try { + const full = Array.from({ length: 20 }, (_, i) => ({ agentName: 'prime', turnIndex: i, sequenceBefore: i * 3, branchName: `b${i}`, timestamp: i })); + checkpoints.saveTurnCheckpoints('prime', full.map((c) => ({ ...c }))); + checkpoints.turnCounters.set('prime', 20); + t.send('Are you there?'); + assert.ok(await pollUntil(() => t.of('inference:held').length === 1, 5_000)); + backlog.fit = true; + assert.ok(await turnOver(t)); + const kept = checkpoints.getTurnCheckpoints('prime'); + assert.deepEqual(kept.slice(0, 19), full.slice(1), 'the nineteen newest from before the turn, each as it was'); + assert.equal(kept.length, 20); + assert.equal(kept[19]!.turnIndex, 20, 'and the turn\'s own, with the index it took'); + } finally { + await t.close(); + } + }); + }); + + it('invalidates redo at a held start, as at any turn\'s first start', async () => { + await withBacklog({ fit: true }, async (backlog) => { + const t = await residentTurn({ maintenanceIntervalMs: 3_600_000 }); + try { + t.send('A turn that fits'); + assert.ok(await turnOver(t)); + assert.equal(t.framework.undoLastTurn('prime').undone, true); + const before = t.framework.getUndoRedoState('prime'); + assert.equal(before.redoDepth, 1, 'the undone turn can be redone'); + backlog.fit = false; + t.send('Are you there?'); + assert.ok(await pollUntil(() => t.of('inference:held').length === 1, 5_000)); + assert.deepEqual(t.framework.getUndoRedoState('prime'), { ...before, canUndo: true, undoDepth: before.undoDepth + 1, canRedo: false, redoDepth: 0 }, + 'the held start invalidated redo and its turn is one undo step'); + } finally { + await t.close(); + } + }); + }); + + it('ends a held turn whose stop comes as its hold is reported', async () => { + await withBacklog({}, async () => { + const t = await residentTurn({ maintenanceIntervalMs: 3_600_000 }); + let stopped: boolean | undefined; + t.framework.onTrace((e) => { + if (e.type === 'inference:held' && stopped === undefined) stopped = t.framework.abortInference('prime'); + }); + try { + t.send('Are you there?'); + assert.ok(await pollUntil(() => t.of('inference:hold_ended').length === 1, 5_000)); + assert.equal(stopped, true, 'the stop is reported taken'); + assert.deepEqual(t.of('inference:hold_ended').map((e) => e.outcome), ['stopped']); + assert.equal(t.of('inference:aborted').length, 1); + assert.equal(t.internal.budgetHolds.size, 0); + assert.equal(t.framework.getHostModeStatus().drained, true); + assert.deepEqual(t.of('ops:alert').filter((e) => e.kind === 'context-refusal'), [], 'no alert says the ended turn is held'); + } finally { + await t.close(); + } + }); + }); + + it('ends a released turn whose stop comes as its hold\'s end is reported, and runs it no more', async () => { + await withBacklog({ fitAt: 12 }, async () => { + const t = await residentTurn({ maintenanceIntervalMs: 50, responses: 2 }); + let stopped: boolean | undefined; + t.framework.onTrace((e) => { + if (e.type === 'inference:hold_ended' && stopped === undefined) stopped = t.framework.abortInference('prime'); + }); + try { + t.send('Are you there?'); + assert.ok(await pollUntil(() => t.of('inference:aborted').length === 1, 5_000)); + assert.equal(stopped, true, 'the stop is reported taken'); + assert.deepEqual(t.of('inference:hold_ended').map((e) => e.outcome), ['fit']); + assert.equal(t.internal.budgetHolds.size, 0); + t.send('Still there?'); + assert.ok(await pollUntil(() => t.of('inference:completed').length === 1, 5_000), 'a later message gets its turn'); + assert.equal(t.membrane.calls.length, 1, 'and the stopped turn never ran'); + assert.equal(t.of('inference:aborted').length, 1); + } finally { + await t.close(); + } + }); + }); + + it('leaves no record of a released turn whose compile a stop or a disposal reaches', async () => { + for (const end of ['stop', 'disposal'] as const) { + let open!: () => void; + const until = new Promise((resolve) => { open = resolve; }); + await withBacklog({ compileGate: { nth: 2, until } }, async (backlog) => { + const t = await residentTurn({ maintenanceIntervalMs: 50 }); + let closed = false; + try { + t.send('Are you there?'); + assert.ok(await pollUntil(() => t.of('inference:held').length === 1, 5_000)); + backlog.fit = true; + assert.ok(await pollUntil(() => backlog.compiles === 2, 3_000), 'the hold released the turn into its compile'); + if (end === 'stop') { + const stopping = t.close(); + open(); + await stopping; + closed = true; + } else { + t.internal.disposeConversationAgent('prime'); + open(); + assert.ok(await pollUntil(() => t.of('inference:aborted').length === 1, 3_000), 'the compile ended with the agent gone'); + } + assert.equal(t.internal.budgetHolds.size, 0, `nothing of the hold is left after the ${end}`); + } finally { + open(); + if (!closed) await t.close(); + } + }); + } + }); + + it('refuses a puppeted tool call while a turn is held, naming the hold', async () => { + await withBacklog({}, async () => { + const t = await residentTurn({ maintenanceIntervalMs: 3_600_000 }); + try { + t.send('Are you there?'); + assert.ok(await pollUntil(() => t.of('inference:held').length === 1, 5_000)); + await assert.rejects(t.framework.puppetToolCall('prime', 'inbox--anything', {}), + /puppet refused: agent prime is idle\+turn-alive \(a budget hold keeps its turn to start again once its context fits/); + assert.equal(heldRequests(t.framework), 1, 'the turn is still held'); + } finally { + await t.close(); + } + }); + }); +}); + +describe('the release check takes context-manager\'s verdict on the real strategy (#304)', () => { + async function realResident(adaptiveResolution: boolean, withStrategy?: (strategy: AutobiographicalStrategy) => void) { + const proto = ContextManager.prototype as unknown as { + tick: () => Promise; previewContext: (...args: unknown[]) => unknown; compile: (...args: unknown[]) => unknown; + }; + const tick = proto.tick; + proto.tick = async () => {}; + const previewContext = proto.previewContext; + const previewBudgets: unknown[] = []; + proto.previewContext = function (this: unknown, ...args: unknown[]) { + previewBudgets.push(args[0]); + return previewContext.apply(this, args); + }; + const compile = proto.compile; + const compileBudgets: unknown[] = []; + proto.compile = function (this: unknown, ...args: unknown[]) { + if ((args[2] as { dryRun?: boolean } | undefined)?.dryRun !== true) compileBudgets.push(args[0]); + return compile.apply(this, args); + }; + const tempDir = mkdtempSync(join(tmpdir(), 'budget-hold-real-')); + const membrane = new SequentialStreamMembrane(); + membrane.pushResponse(createMockResponse([{ type: 'text', text: 'Answered once it fit' }])); + const strategy = new AutobiographicalStrategy({ adaptiveResolution, targetChunkTokens: 300, recentWindowTokens: 400 }); + withStrategy?.(strategy); + const framework = await AgentFramework.create({ + storePath: join(tempDir, 'store'), + membrane: membrane.asMembrane(), + agents: [{ + name: 'prime', model: 'test-model', systemPrompt: 'You are prime.', contextBudgetTokens: 400_000, maxTokens: 1_000, + strategy, + }], + modules: [], + syncIntervalMs: 0, + maintenanceIntervalMs: 50, + errorPolicy: { maxRetries: 3, onInferenceError: (_error: Error, _agent: string, attempt: number) => (attempt < 3 ? { retry: true, delayMs: 0 } : { retry: false }) }, + }); + const traces: TraceEvent[] = []; + framework.onTrace((e) => traces.push(e)); + const cm = framework.getAgent('prime')!.getContextManager(); + for (let i = 0; i < 16; i++) { + cm.addMessage(i % 2 === 0 ? 'User' : 'prime', [{ type: 'text', text: `m${i} ` + 'lorem ipsum dolor sit amet '.repeat(i % 3 === 0 ? 500 : 60) }]); + } + const budget = (maxTokens: number) => ({ maxTokens, reserveForResponse: 1_000 }); + return { + framework, + membrane, + of: (type: string) => traces.filter((e) => e.type === type) as unknown as Array>, + preview: (maxTokens: number) => (cm as unknown as { + previewContext: (b: { maxTokens: number; reserveForResponse: number }) => { fits: boolean; finalTokens: number; budgetTokens: number }; + }).previewContext(budget(maxTokens)), + addMessage: (text: string) => cm.addMessage('User', [{ type: 'text', text }]), + dryCompileResolves: (maxTokens: number) => cm.compile(budget(maxTokens), undefined, { dryRun: true } as never).then(() => true, () => false), + setBudget: (tokens: number) => framework.updateAgentRuntimeSettings('prime', { contextBudgetTokens: tokens, immediate: true }, { allowInfeasible: true }), + wake: () => (framework as unknown as { pendingRequests: unknown[] }).pendingRequests.push({ + agentName: 'prime', reason: 'external-message', source: 'test', timestamp: Date.now(), + }), + previews: () => previewBudgets.length, + previewBudgets, + compileBudgets, + close: async () => { + await framework.stop(); + proto.tick = tick; + proto.previewContext = previewContext; + proto.compile = compile; + rmSync(tempDir, { recursive: true, force: true }); + }, + }; + } + + it('holds a released turn again where the note its start stores tips its compile over, and runs it once there is room', async () => { + const t = await realResident(true); + try { + let lo = 2_000; + let hi = 400_000; + assert.equal(t.preview(lo).fits, false); + while (hi - lo > 1) { + const mid = Math.floor((lo + hi) / 2); + if (t.preview(mid).fits) hi = mid; else lo = mid; + } + const atHi = t.preview(hi); + const spare = atHi.budgetTokens - atHi.finalTokens; + assert.ok(spare >= 0 && spare < 10, + `${spare} tokens to spare at ${hi} (a plan of ${atHi.finalTokens} against ${atHi.budgetTokens}), well under the note's size`); + t.setBudget(lo); + t.framework.start(); + const before = t.previews(); + t.wake(); + assert.ok(await pollUntil(() => t.of('inference:held').length === 1, 5_000), 'the turn\'s compile is refused below the plan'); + assert.ok(await pollUntil(() => t.previews() > before, 3_000), 'and the release check has looked'); + assert.deepEqual(t.previewBudgets.at(-1), t.compileBudgets.at(-1), 'at the budget the refused compile planned at'); + t.setBudget(hi); + assert.ok(await pollUntil(() => t.of('inference:held').length === 2, 5_000), + 'that budget releases it, and its compile, with the note stored, is refused at the margin'); + assert.equal(t.membrane.calls.length, 0); + await new Promise((r) => setTimeout(r, 500)); + assert.equal(t.of('inference:held').length, 2, 'the stored note brings no release of its own'); + t.setBudget(hi + 1_000); + assert.ok(await pollUntil(() => storedText(t.framework).includes('Answered once it fit'), 5_000), 'a budget with room for the note runs the turn'); + assert.deepEqual(t.of('inference:hold_ended').map((e) => e.outcome), ['fit', 'fit']); + assert.equal(JSON.stringify(t.membrane.calls[0]).split('[budget-hold]').length - 1, 2, + 'its compile reads the note each release stored'); + } finally { + await t.close(); + } + }); + + it('holds a context refusal once on the preview path, and runs the turn once the refusal clears', async () => { + // A coverage defect in the strategy's emission, one emitted entry left out + // while `drop` is set, makes its coverage check refuse every select with + // UncoveredDropError: the committing compile's and the preview's alike. + // Context-manager's preview rethrows that refusal rather than answering + // fits: false. + let drop = true; + const t = await realResident(true, (strategy) => { + const s = strategy as unknown as { mergeAdjacentBodyGroupRaw: (...args: unknown[]) => unknown[] }; + const merge = s.mergeAdjacentBodyGroupRaw.bind(strategy); + s.mergeAdjacentBodyGroupRaw = (...args: unknown[]) => { + const out = merge(...args); + if (drop && out.length > 2) out.splice(1, 1); + return out; + }; + }); + try { + assert.throws(() => t.preview(400_000), (e: unknown) => (e as { name?: string }).name === 'UncoveredDropError', + 'the preview rethrows the context refusal'); + t.framework.start(); + t.wake(); + assert.ok(await pollUntil(() => t.of('inference:held').length === 1, 5_000)); + assert.equal(t.of('inference:held')[0]!.errorType, 'context_refusal'); + for (let i = 0; i < 4; i++) { + const looked = t.previews(); + t.addMessage(`later ${i}`); + assert.ok(await pollUntil(() => t.previews() > looked, 3_000), `a look after message ${i + 1}`); + } + assert.equal(t.of('inference:held').length, 1, 'held once while the refusal stands, through those looks'); + assert.deepEqual(t.of('inference:hold_ended'), []); + assert.equal(t.of('ops:alert').filter((e) => e.kind === 'context-refusal').length, 1); + assert.equal(t.membrane.calls.length, 0); + drop = false; + t.addMessage('after the defect'); + assert.ok(await pollUntil(() => storedText(t.framework).includes('Answered once it fit'), 5_000), 'the turn runs once the refusal clears'); + assert.deepEqual(t.of('inference:hold_ended').map((e) => e.outcome), ['fit']); + } finally { + await t.close(); + } + }); + + it('falls back to a dry compile where the preview needs adaptive resolution', async () => { + const t = await realResident(false); + try { + // The hierarchical strategy refuses an over-budget context in its + // preview and in a dry compile alike. A context that fits has no fold + // plan to preview: context-manager 0.23.0's preview throws, and from + // 0.23.1 it returns null. The dry compile decides. + let fitsFrom = 10_000; + while (!(await t.dryCompileResolves(fitsFrom)) && fitsFrom < 400_000) fitsFrom += 1_000; + assert.ok(fitsFrom < 400_000); + assert.equal(await t.dryCompileResolves(Math.round(fitsFrom * 0.8)), false, + 'its dry compile refuses where the context does not fit, as a committing compile does'); + let verdict: unknown; + try { + verdict = t.preview(fitsFrom); + } catch (error) { + assert.match((error as Error).message, /requires adaptiveResolution/); + verdict = null; + } + assert.equal(verdict, null, 'the preview gives no verdict for a context that fits'); + t.setBudget(Math.round(fitsFrom * 0.8)); + t.framework.start(); + const before = t.previews(); + t.wake(); + assert.ok(await pollUntil(() => t.of('inference:held').length === 1, 5_000)); + assert.ok(await pollUntil(() => t.previews() > before, 3_000), 'the release check has looked'); + await new Promise((r) => setTimeout(r, 500)); + assert.deepEqual(t.of('inference:hold_ended'), [], 'it stays held while the context does not fit'); + t.setBudget(fitsFrom + 2_000); + assert.ok(await pollUntil(() => storedText(t.framework).includes('Answered once it fit'), 5_000)); + assert.deepEqual(t.of('inference:hold_ended').map((e) => e.outcome), ['fit'], 'the dry compile said it fits'); + } finally { + await t.close(); + } + }); +}); diff --git a/test/stop-step-throws.test.ts b/test/stop-step-throws.test.ts index c1f657c4a..911e71590 100644 --- a/test/stop-step-throws.test.ts +++ b/test/stop-step-throws.test.ts @@ -166,6 +166,25 @@ const ROWS: Row[] = [ }, }, { step: "cancelling provider admission", inject: (fw, e) => patchMethod(fw, 'cancelProviderAdmission', e) }, + { + step: "a budget hold's drop", + inject: (fw, e) => { + // Two turns' hold records: the first one's drop throws. + const holds = internals(fw).budgetHolds as Map; + holds.set('first', { state: 'released', heldRequests: [] }); + holds.set('second', { state: 'released', heldRequests: [] }); + const owner = fw as unknown as Record; + const usual = (owner.dropBudgetHold as (name: string) => void).bind(fw); + owner.dropBudgetHold = (name: string) => { + if (name === 'first') throw e; + usual(name); + }; + return () => { delete owner.dropBudgetHold; }; + }, + check: (fw) => { + assert.equal((internals(fw).budgetHolds as Map).has('second'), false, "the other hold's drop still ran"); + }, + }, { step: "a stream's waiting outgoing complete", inject: (fw, e) => {