From b0a8a531e13a5332c555c6fc5bd9fdeaca20a9ee Mon Sep 17 00:00:00 2001 From: Gladys-2413 Date: Thu, 15 Oct 2026 08:56:03 -0400 Subject: [PATCH 01/11] fix: the six findings of #352's review Anarchid's review of #352 approved it with six minor findings. This answers each of them. R1-1. The host hide command's range form resolved both ends to the first stored copy of their message, so a range ending at a message stored twice (a write cut short, then retried) left the retry in context. The earlier end now resolves to its first copy and the later end to its last, as offline recovery's range does, before the range widens to whole bodies. R1-2. A conversation agent's maintenance calls go through conversationAuxiliaryMembraneFor, which refused a held model but didn't withdraw a call made while a surgery holds the store, as a resident's withAuxiliaryAdmission does. It now withdraws it, so "Maintenance makes no provider call meanwhile" holds for conversation agents too. R1-3. previewAgentRuntimeSettings threw when the agent's context manager refused its queries, as it does while its strategy's initialization is pending or after it failed, though its contract is that preview-layer reasons come back as {available: false, reason}. It now answers that, with the agent's live budget, the patch's path and the transition, all from the agent's own state. A patch the agent refuses still throws, whatever the manager's state, and any other error still throws. resume()'s catch for that case is gone: an unavailable preview already gives no verdict there. R1-4. The 0.21.1 operator note's multiplier reset now says which stores it is for: one that already opened on context-manager 0.14.0 and ran a committing compile. ^0.14.0 now installs 0.14.1, which keeps a still-unstamped multiplier provisional and replaces or drops it on its own. R1-5. A maintenance failure that lasts is met by every pass, and each pass went into the 50-entry maintenance history, so within minutes it pushed out the runs that did work. A pass whose every record repeats a failure already reported, with its report suppressed, now stays out of the history. R1-6. stop() awaited only the maintenance pass, so an over-budget drain kick's compression could still be running when the store closed, and the kick went on ticking after stop() began. stop() now waits for the pass and its drain kicks, and a kick starts no new tick once the framework is stopping. Tests: the range case in operator-surgery's body test; a conversation agent's tick withdrawn under a hold, the history test and a stop() test in maintenance-timer; the preview's answer under a refusing manager in host-quiesce. Two maintenance-timer tests that waited for the history to grow with each repeated failure now count passes, and harnesses that build the framework without start() set running. --- CHANGELOG.md | 5 +- changelog.d/body-removals.fixed.md | 2 +- changelog.d/cm-0.14.changed.md | 4 +- changelog.d/cm-0.14.fixed.md | 2 +- src/agent.ts | 50 +++++++- src/framework.ts | 95 +++++++++------ test/host-quiesce.test.ts | 19 ++- test/inference-failure-bounds.test.ts | 2 + test/maintenance-timer.test.ts | 167 +++++++++++++++++++++++++- test/operator-surgery.test.ts | 26 ++++ test/overbudget-drain-breaker.test.ts | 2 + 11 files changed, 318 insertions(+), 56 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index a7cf31a15..ffa091bd5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -711,7 +711,10 @@ framework. below 1, which its `reset.discardedMultiplier` shows. With the host stopped, reset the first kind's multiplier once, and re-stamp the second kind's checked value, as the **Operators** entry of context-manager - 0.14.0's changelog describes. + 0.14.0's changelog describes. The reset is for a store that already + opened on 0.14.0 and ran a committing compile, which re-stamped its + multiplier: `^0.14.0` now installs 0.14.1, which keeps a still-unstamped + multiplier provisional and replaces or drops it on its own. - **What else hosts meet with context-manager 0.14.0** (added after the release): a preview, or any compile made with `dryRun`, leaves the context manager's estimate, its calibration and a pending kv-unified draft's marker diff --git a/changelog.d/body-removals.fixed.md b/changelog.d/body-removals.fixed.md index d71c3d50a..a0db3c730 100644 --- a/changelog.d/body-removals.fixed.md +++ b/changelog.d/body-removals.fixed.md @@ -1 +1 @@ -- Removing a single message that belongs to a body, a one-shard body's included, no longer fails: context-manager's `removeMessage` refuses any message in a body, so agent-framework now removes one as a range in offline recovery's suppressions, and in a Discord suppression resumed after a crash. The host `hide` command now hides a message stored as a body (a long one, in shards) whole, with every stored copy of it (a write cut short leaves a partial copy before its retry), and its operator-log entry lists at most 50 of the ids it removed, with `removedIdsTruncated` when there were more. It's refused as busy while a surgery holds the store, and a message with no Discord id never matches one. A module's removal of a message it added, and the coalescer's, take its whole body too, and an automatic rewind whose newest message is a shard of a body takes the whole body. +- Removing a single message that belongs to a body, a one-shard body's included, no longer fails: context-manager's `removeMessage` refuses any message in a body, so agent-framework now removes one as a range in offline recovery's suppressions, and in a Discord suppression resumed after a crash. The host `hide` command now hides a message stored as a body (a long one, in shards) whole, with every stored copy of it (a write cut short leaves a partial copy before its retry). Its range form runs from the earlier message's first copy through the later message's last, as offline recovery's range does. Either form's operator-log entry lists at most 50 of the ids it removed, with `removedIdsTruncated` when there were more. It's refused as busy while a surgery holds the store, and a message with no Discord id never matches one. A module's removal of a message it added, and the coalescer's, take its whole body too, and an automatic rewind whose newest message is a shard of a body takes the whole body. diff --git a/changelog.d/cm-0.14.changed.md b/changelog.d/cm-0.14.changed.md index 8e997f2ea..7ac6469b1 100644 --- a/changelog.d/cm-0.14.changed.md +++ b/changelog.d/cm-0.14.changed.md @@ -1,2 +1,2 @@ -- A context-maintenance pass no longer fails as a whole when one agent's readiness can't be read. context-manager 0.14, which agent-framework has required since 0.21.1, refuses its readiness queries while a strategy's initialization is pending (inside `switchBranch`, `fork`, `setStrategy` or `ContextManager.open`) or has failed, and a strategy can refuse on its own, as the autobiographical one does for a branch it hasn't reloaded. That agent's run now records the error, the agent gets its own `context-maintenance-failed` alert (`scope: 'agent'`), and the other agents run. Before, such a throw ended the pass for every agent, with one alert for the whole pass (`scope: 'pass'`). Anything else that throws while one agent's maintenance is prepared is that agent's failure in the same way. A failure that lasts, met by every pass, is recorded on each run, and logged and alerted once, then again when its text changes, every 15 minutes while it lasts (the alert webhook's cooldown for it, counted from the webhook's last post of it), or after that agent's maintenance has gone without a failure. `getContextMaintenanceSnapshot()` reports such an agent as `ready: false` with an `error`, and a strategy that can't give a progress snapshot reads as having none. -- While an operator surgery (`rollbackToMessage` or `suppressMessages`) or `switchBranch` holds the store, no maintenance pass starts and no over-budget drain kick is started, a running pass or drain kick starts no new tick, and the operator call waits for them to finish before it switches or forks the branch, so no maintenance work overlaps the move. Maintenance makes no provider call meanwhile: one made then, or one waiting at the agent's provider gate when the hold is taken, is withdrawn without being sent. That's no maintenance failure, and context-manager counts no attempt for it. A merge keeps its place in the queue. A chunk's compression logs context-manager's "Failed to compress chunk (hierarchical)" line with the withdrawal, and the chunk is queued again at that manager's next rebuild, such as its next message or compile. The operator call never waits on a provider cooldown, or on the retry turn a cooldown queues, which the hold keeps from starting. `switchBranch` can therefore move the store after the call returns, and the API server's `branch.switch` announces the move it made once it has, while `branch.current`, the branch list, `/health` and `/status` read the store's branch whenever they report one. `maintenanceTick()` answers `ran: false` meanwhile, and `resume()` refuses with `OperatorActionError` (`agent-busy`), as the other operator calls do, including when the surgery took the store while `resume()` waited. Resume once the surgery has finished. +- A context-maintenance pass no longer fails as a whole when one agent's readiness can't be read. context-manager 0.14, which agent-framework has required since 0.21.1, refuses its readiness queries while a strategy's initialization is pending (inside `switchBranch`, `fork`, `setStrategy` or `ContextManager.open`) or has failed, and a strategy can refuse on its own, as the autobiographical one does for a branch it hasn't reloaded. That agent's run now records the error, the agent gets its own `context-maintenance-failed` alert (`scope: 'agent'`), and the other agents run. Before, such a throw ended the pass for every agent, with one alert for the whole pass (`scope: 'pass'`). Anything else that throws while one agent's maintenance is prepared is that agent's failure in the same way. A failure that lasts, met by every pass, is recorded in the maintenance history once, and logged and alerted once, then again when its text changes, every 15 minutes while it lasts (the alert webhook's cooldown for it, counted from the webhook's last post of it), or after that agent's maintenance has gone without a failure; a pass that only met failures already reported isn't added to the history, so a lasting one no longer pushes out the runs that did work. `getContextMaintenanceSnapshot()` reports such an agent as `ready: false` with an `error`, and a strategy that can't give a progress snapshot reads as having none. +- While an operator surgery (`rollbackToMessage` or `suppressMessages`) or `switchBranch` holds the store, no maintenance pass starts and no over-budget drain kick is started, a running pass or drain kick starts no new tick, and the operator call waits for them to finish before it switches or forks the branch, so no maintenance work overlaps the move. `stop()` waits for them the same way, a drain kick included, before it closes the store, and a drain kick starts no new tick once `stop()` has begun. Maintenance makes no provider call meanwhile, a conversation agent's included: one made then, or one waiting at the agent's provider gate when the hold is taken, is withdrawn without being sent. That's no maintenance failure, and context-manager counts no attempt for it. A merge keeps its place in the queue. A chunk's compression logs context-manager's "Failed to compress chunk (hierarchical)" line with the withdrawal, and the chunk is queued again at that manager's next rebuild, such as its next message or compile. The operator call never waits on a provider cooldown, or on the retry turn a cooldown queues, which the hold keeps from starting. `switchBranch` can therefore move the store after the call returns, and the API server's `branch.switch` announces the move it made once it has, while `branch.current`, the branch list, `/health` and `/status` read the store's branch whenever they report one. `maintenanceTick()` answers `ran: false` meanwhile, and `resume()` refuses with `OperatorActionError` (`agent-busy`), as the other operator calls do, including when the surgery took the store while `resume()` waited. Resume once the surgery has finished. diff --git a/changelog.d/cm-0.14.fixed.md b/changelog.d/cm-0.14.fixed.md index a4e121c59..dc016c453 100644 --- a/changelog.d/cm-0.14.fixed.md +++ b/changelog.d/cm-0.14.fixed.md @@ -1,2 +1,2 @@ -- `resume()` no longer throws when one agent's runtime settings can't be read, as when its context manager (0.14, required since 0.21.1) refuses after its strategy failed to initialize. It warns, naming the agent and the error, and proceeds without a feasibility verdict for that agent: the verdict guards against an agent that would run over budget, and an agent whose manager refuses its compiles can't run. That manager refuses the agent's turns and writes until a `switchBranch` or `setStrategy` succeeds, and maintenance reports it. +- `previewAgentRuntimeSettings()` no longer throws when an agent's context manager (0.14, required since 0.21.1) refuses its queries, as after its strategy failed to initialize: it answers `{available: false, reason}`, with the agent's live budget and the patch's path, as it does for any preview it can't compute. So `resume()` no longer throws for such an agent either. It warns, naming the agent and the reason, and proceeds without a feasibility verdict for that agent: the verdict guards against an agent that would run over budget, and an agent whose manager refuses its compiles can't run. That manager refuses the agent's turns and writes until a `switchBranch` or `setStrategy` succeeds, and maintenance reports it. - Offline recovery (`createOfflineRecoveryBranch`) reads a sharded message's body as context-manager 0.14.0 does: a message continues the body before it only when it carries the same group id at a higher shard index. A document posted twice stores two copies under one group id, so an anchor at the end of one copy, or a suppression of one copy alone, is no longer refused. A selection inside one copy is still refused. An anchor at the end of a write that was cut short keeps that partial body and leaves out its retry, which context-manager 0.14.0 reads as a body of its own. diff --git a/src/agent.ts b/src/agent.ts index 9780d879d..117fda103 100644 --- a/src/agent.ts +++ b/src/agent.ts @@ -496,6 +496,40 @@ export class Agent { return result; } + /** + * The budget compiles plan at, the path a patch's budget takes from it, and + * whether a descent is in flight, from the agent's own state alone: what a + * runtime-settings preview can still say while the context manager refuses + * its queries. + */ + runtimeBudgetPath(patch?: AgentRuntimeSettingsPatch): { + liveBudgetTokens: number; + path: 'immediate' | 'paced' | 'none'; + transition: 'stable' | 'converging'; + } { + const live = this.contextBudgetTokens ?? DEFAULT_CONTEXT_BUDGET_TOKENS; + const budget = patch?.contextBudgetTokens; + return { + liveBudgetTokens: live, + path: budget === undefined ? 'none' : budget >= live || patch?.immediate ? 'immediate' : 'paced', + transition: this.contextBudgetTargetTokens === undefined ? 'stable' : 'converging', + }; + } + + /** + * Whether the context manager answers its queries. It refuses them while + * its strategy's initialization is pending or has failed, until a + * switchBranch or setStrategy succeeds. + */ + contextManagerAnswers(): boolean { + try { + this.contextManager.isReady(); + return true; + } catch { + return false; + } + } + /** * Read-only mirror of `updateRuntimeSettings`' budget semantics: derive what * the next compile would actually plan at if `patch` were applied, WITHOUT @@ -538,9 +572,8 @@ export class Agent { this.validateRuntimeSettingsPatch(patch); } const hot = this.getHotContextSettings(); - const live = this.contextBudgetTokens ?? DEFAULT_CONTEXT_BUDGET_TOKENS; + const { liveBudgetTokens: live, path } = this.runtimeBudgetPath(patch); const budget = patch?.contextBudgetTokens; - const path = budget === undefined ? 'none' : budget >= live || patch?.immediate ? 'immediate' : 'paced'; const overrides: Record = {}; if (patch?.tailTokens !== undefined) { @@ -757,6 +790,15 @@ export class Agent { } private validateRuntimeSettingsPatch(patch: AgentRuntimeSettingsPatch): void { + this.checkRuntimeSettingsPatch(patch); + if ((patch.tailTokens !== undefined || patch.transitionPaceTokens !== undefined) && + !this.getHotContextSettings()) { + throw new Error('The active context strategy does not support live tail/transition settings'); + } + } + + /** The checks of a runtime-settings patch that need nothing from the context manager. */ + checkRuntimeSettingsPatch(patch: AgentRuntimeSettingsPatch): void { if (Object.keys(patch).length === 0) throw new Error('At least one setting is required'); if (patch.contextBudgetTokens !== undefined) { if (!Number.isSafeInteger(patch.contextBudgetTokens) || patch.contextBudgetTokens <= this.maxTokens) { @@ -776,10 +818,6 @@ export class Agent { patch.sameRoundThinkTextPolicy !== 'private') { throw new Error("sameRoundThinkTextPolicy must be 'public' or 'private'"); } - if ((patch.tailTokens !== undefined || patch.transitionPaceTokens !== undefined) && - !this.getHotContextSettings()) { - throw new Error('The active context strategy does not support live tail/transition settings'); - } } /** Settle a converging budget once the window is prepared; true if it settled. */ diff --git a/src/framework.ts b/src/framework.ts index 919f722fe..b2325f51f 100644 --- a/src/framework.ts +++ b/src/framework.ts @@ -1558,6 +1558,8 @@ export class AgentFramework { * it was alerted, until its maintenance goes without one * (reportMaintenanceFailure). */ private maintenanceFailureReported = new Map(); + /** Run records whose failure repeated one already reported, and whose report was suppressed. */ + private maintenanceRepeats = new WeakSet(); private providerGates: Map = new Map(); private providerAccelerationCooldowns: Map = new Map(); private providerAccelerationRecoveries: Map = new Map(); @@ -2693,9 +2695,9 @@ export class AgentFramework { clearInterval(this.maintenanceTimer); this.maintenanceTimer = null; } - if (this.maintenancePass) { - await this.maintenancePass; - } + // The pass and any over-budget drain kick: a compression in flight lands + // before the store closes. + await this.settleMaintenancePass(); if (this.loopPromise) { await this.loopPromise; @@ -2896,11 +2898,14 @@ export class AgentFramework { * as a resident's are, so they bind its auxiliary calls too: a held model * is refused without a call, and an auxiliary call's own stated wait is * recorded for it. (An ephemeral run records nothing, its name being - * single-use, so its context manager keeps the plain membrane.) + * single-use, so its context manager keeps the plain membrane.) A + * conversation agent is ticked by the maintenance pass too, so a call made + * while a surgery holds the store is withdrawn, as a resident's is. */ private conversationAuxiliaryMembraneFor(agentName: string): Membrane { return this.membraneAdmitting(async (run, model) => { this.refuseHeldProviderModel(agentName, model); + this.withdrawMaintenanceUnderSurgery(); try { return await run(); } catch (error) { if (model !== undefined && error instanceof Error) this.recordProviderWait(agentName, model, error); @@ -3532,7 +3537,10 @@ export class AgentFramework { run.finishedAt = Date.now(); run.agents.sort((a, b) => a.agentName.localeCompare(b.agentName)); this.currentMaintenanceRun = null; - this.maintenanceHistory.unshift(run); + // A pass that only met failures already reported adds nothing the + // history doesn't hold, and a lasting one would otherwise fill it in + // minutes, pushing out the runs that did work. + if (!run.agents.every((record) => this.maintenanceRepeats.has(record))) this.maintenanceHistory.unshift(run); if (this.maintenanceHistory.length > 50) this.maintenanceHistory.length = 50; } } @@ -3558,7 +3566,10 @@ export class AgentFramework { // changed text that the webhook didn't post doesn't push the reminder // back. With no webhook, it's counted from this failure's last report. const since = this.opsAlertLastSent.get(`${agentName}:context-maintenance-failed`) ?? reported.at; - if (now - since < this.opsAlertCooldownMs) return; + if (now - since < this.opsAlertCooldownMs) { + this.maintenanceRepeats.add(record); + return; + } } this.maintenanceFailureReported.set(agentName, { text: wholeError, at: now }); console.error(`[context-maintenance] agent=${agentName} failed:`, record.error); @@ -4674,26 +4685,16 @@ export class AgentFramework { if (this.maintenanceRunning()) await this.settleMaintenancePass(); const failing: Array<{ agentName: string; preview: RuntimeSettingsPreview }> = []; for (const agentName of this.agents.keys()) { - let preview: RuntimeSettingsPreview; - try { + // A preview that isn't available gives no verdict, and resume goes on + // for the others. That includes an agent whose context manager refuses + // its queries, as when its strategy failed to initialize: that manager + // refuses its compiles and writes until a switchBranch or setStrategy + // succeeds, so the agent can't run over budget, and maintenance reports it. + let preview = this.previewAgentRuntimeSettings(agentName); + for (let attempt = 0; attempt < 8 && !preview.available && preview.reason === 'preview_in_flight'; attempt++) { + await new Promise((resolve) => setTimeout(resolve, 250)); + if (this.maintenancePass) await this.maintenancePass.catch(() => {}); preview = this.previewAgentRuntimeSettings(agentName); - for (let attempt = 0; attempt < 8 && !preview.available && preview.reason === 'preview_in_flight'; attempt++) { - await new Promise((resolve) => setTimeout(resolve, 250)); - if (this.maintenancePass) await this.maintenancePass.catch(() => {}); - preview = this.previewAgentRuntimeSettings(agentName); - } - } catch (error) { - // The agent's settings can't be read, as when its context manager's - // strategy failed to initialize: that manager refuses its compiles and - // writes until a switchBranch or setStrategy succeeds, and maintenance - // reports it. The gate guards against an agent that would run over - // budget, and this one can't run, so it gets no verdict, as a preview - // that isn't available gets none, and resume goes on for the others. - console.warn( - `[host-mode] resume: ${agentName}'s settings can't be read ` + - `(${error instanceof Error ? error.message : String(error)}) — proceeding without a verdict for it`, - ); - continue; } if (preview.available && preview.effective && !preview.effective.fits) { failing.push({ agentName, preview }); @@ -5295,9 +5296,11 @@ export class AgentFramework { * the compile budget and rejected safe-by-construction paced descents. * * Never throws for preview-layer reasons: strategy without previewContext, - * preview overlap, stale branch generation, and the hierarchical (non- - * adaptive) path all come back as `{available: false, reason}`. Unknown - * agent still throws, matching the sibling accessors. + * preview overlap, stale branch generation, the hierarchical (non- + * adaptive) path, and a context manager that refuses its queries (its + * strategy's initialization pending or failed) all come back as + * `{available: false, reason}`. Unknown agent still throws, matching the + * sibling accessors, and so does a patch the agent refuses. */ previewAgentRuntimeSettings( agentName: string, @@ -5313,8 +5316,20 @@ export class AgentFramework { agent: Agent, patch?: AgentRuntimeSettingsPatch, ): RuntimeSettingsPreview { - const plan = agent.planRuntimeSettings(patch); - const settings = agent.getRuntimeSettings(); + // A patch the agent refuses throws, whatever its context manager's state. + if (patch && Object.keys(patch).length > 0) agent.checkRuntimeSettingsPatch(patch); + let plan: ReturnType; + let settings: ReturnType; + try { + plan = agent.planRuntimeSettings(patch); + settings = agent.getRuntimeSettings(); + } catch (error) { + // A manager that refuses its queries gives no verdict, as a preview + // that can't be computed gives none: the agent's own state still says + // its budget and the patch's path. Anything else throws as before. + if (agent.contextManagerAnswers()) throw error; + return { available: false, reason: error instanceof Error ? error.message : String(error), ...agent.runtimeBudgetPath(patch) }; + } const base: RuntimeSettingsPreview = { available: false, path: plan.path, @@ -6983,6 +6998,10 @@ export class AgentFramework { return id === undefined || id === null || id === '' ? undefined : String(id); }; const byDiscordId = (did: string) => all.findIndex((m) => discordIdOf(m) === String(did)); + const lastByDiscordId = (did: string) => { + for (let i = all.length - 1; i >= 0; i--) if (discordIdOf(all[i]) === String(did)) return i; + return -1; + }; // Collect Discord refs of every message in [lo, hi] that carries one, // so the surface can mark them. channelId here is whatever the ingest @@ -7028,7 +7047,13 @@ export class AgentFramework { if (toIdx < 0) { return { ok: false, error: `Message ${params.toMessageId} is not an addressable message in context.` }; } - const [first, last] = fromIdx <= toIdx ? [fromIdx, toIdx] : [toIdx, fromIdx]; + // The earlier end from its first stored copy, the later end through + // its last, as offline recovery's range does: a write cut short and + // retried stores a message twice, and the range takes both copies. + const [first, laterId] = fromIdx <= toIdx + ? [fromIdx, params.toMessageId] + : [toIdx, params.fromMessageId]; + const last = lastByDiscordId(laterId); // A message stored as a body (a long one, in shards) is hidden whole. const lo = bodyRangeIn(all, first).from; const hi = bodyRangeIn(all, last).to; @@ -15733,14 +15758,12 @@ export class AgentFramework { let ticks = 0; try { const cm = agent.getContextManager(); - while (ticks < 8 && !this.surgeryHold) { + while (ticks < 8 && this.running && !this.surgeryHold) { await cm.tick(); ticks++; } - console.error( - `[inference-failed] drain kicked for ${agentName} (OverBudget breaker, ${ticks} ticks` + - `${ticks < 8 ? ', stopped while a surgery holds the store' : ''})`, - ); + const stopped = ticks >= 8 ? '' : !this.running ? ', stopped as the framework stops' : ', stopped while a surgery holds the store'; + console.error(`[inference-failed] drain kicked for ${agentName} (OverBudget breaker, ${ticks} ticks${stopped})`); } catch (err) { if (err instanceof MaintenanceWithdrawnError) { console.error(`[inference-failed] drain kicked for ${agentName} (OverBudget breaker, ${ticks} ticks, stopped: ${err.message})`); diff --git a/test/host-quiesce.test.ts b/test/host-quiesce.test.ts index 55213e976..c1fe9680d 100644 --- a/test/host-quiesce.test.ts +++ b/test/host-quiesce.test.ts @@ -329,6 +329,19 @@ test('resume proceeds without a verdict for an agent whose context manager refus await assert.rejects(cm.setStrategy(failing), /strategy initialization failed for the test/); assert.throws(() => cm.getHotContextSettings(), /requires successful strategy initialization/); + // The preview answers rather than throws: no verdict, with what the + // agent's own state still says. + // The agent runs at the default budget of 100,000 tokens, so 50,000 is a paced descent. + const preview = framework.previewAgentRuntimeSettings('agent', { contextBudgetTokens: 50_000 }); + assert.equal(preview.available, false); + assert.match(String(preview.reason), /requires successful strategy initialization/); + assert.equal(preview.path, 'paced'); + assert.equal(preview.liveBudgetTokens, 100_000); + assert.equal(preview.transition, 'stable'); + assert.equal(preview.effective, undefined); + // A patch the agent refuses still throws. + assert.throws(() => framework.previewAgentRuntimeSettings('agent', { contextBudgetTokens: 5 }), /contextBudgetTokens must be/); + const warn = t.mock.method(console, 'warn', () => {}); let status: Awaited>; try { @@ -339,10 +352,10 @@ test('resume proceeds without a verdict for an agent whose context manager refus assert.equal(status.quiesced, false); assert.ok(warn.mock.calls.some((call) => { const line = String(call.arguments[0]); - return line.includes("agent's settings can't be read") + return line.includes('feasibility preview unavailable for agent') && line.includes('requires successful strategy initialization') - && line.includes('proceeding without a verdict for it'); - }), 'resume warns, naming the agent and the error'); + && line.includes('proceeding without a verdict'); + }), 'resume warns, naming the agent and the reason'); }); }); diff --git a/test/inference-failure-bounds.test.ts b/test/inference-failure-bounds.test.ts index 6fe63cdbe..352f6e498 100644 --- a/test/inference-failure-bounds.test.ts +++ b/test/inference-failure-bounds.test.ts @@ -55,6 +55,8 @@ function makeHarness(opts?: { tick?: () => Promise; addMessage?: (p: strin fw.pendingRequests = []; fw.overBudgetDrainInFlight = new Set(); fw.overBudgetDrains = new Set(); + // A drain kick ticks only while the framework runs. + fw.running = true; fw.opsAlertLastSent = new Map(); fw.opsAlertCooldownMs = 15 * 60_000; const traces: any[] = []; diff --git a/test/maintenance-timer.test.ts b/test/maintenance-timer.test.ts index 8b5e6612d..bf6b16139 100644 --- a/test/maintenance-timer.test.ts +++ b/test/maintenance-timer.test.ts @@ -329,6 +329,49 @@ class ParkedCallTickStrategy implements ContextStrategy { } } +/** A strategy whose first tick waits on a gate, then calls its manager's membrane, as a compression would. */ +class CallAfterGateStrategy implements ContextStrategy { + readonly name = 'call-after-gate-test'; + ticks = 0; + entered = false; + outcome?: string; + private open?: () => void; + private readonly gate = new Promise((resolve) => { this.open = resolve; }); + + release(): void { + this.open?.(); + } + + checkReadiness(): ReadinessState { + return { ready: false, description: 'a compression queued' }; + } + + async tick(ctx: StrategyContext): Promise { + this.ticks++; + if (this.ticks > 1) return; + this.entered = true; + await this.gate; + try { + await ctx.membrane?.complete({ + model: 'test-model', + messages: [{ role: 'user', content: [{ type: 'text', text: 'compress' }] }], + } as unknown as NormalizedRequest); + this.outcome = 'sent'; + } catch (error) { + this.outcome = (error as Error).name; + throw error; + } + } + + select( + _store: MessageStoreView, + _log: ContextLogView, + _budget: TokenBudget, + ): ContextEntry[] { + return []; + } +} + /** A strategy whose ticks succeed and whose readiness can't be read once all of a pass's ticks have run. */ class RefusesAfterTicksStrategy implements ContextStrategy { readonly name = 'refuses-after-ticks-test'; @@ -588,11 +631,12 @@ describe('queued context maintenance timer', () => { pending: 'compression blocked', }, }); - // Every pass meets the same failure and records it; it is alerted once. - await waitFor(() => framework.getContextMaintenanceSnapshot().history.length >= 3); - for (const run of framework.getContextMaintenanceSnapshot().history) { - assert.equal(run.agents[0].error, 'provider rejected maintenance request'); - } + // Every pass meets the same failure. It is alerted once, and the + // passes that only repeat it stay out of the history. + await waitFor(() => (framework as unknown as { maintenanceRunId: number }).maintenanceRunId >= 3); + const history = framework.getContextMaintenanceSnapshot().history; + assert.equal(history.length, 1); + assert.equal(history[0]!.agents[0]!.error, 'provider rejected maintenance request'); assert.equal(alerts.length, 1); } finally { await framework.stop(); @@ -801,9 +845,10 @@ describe('queued context maintenance timer', () => { await fence(); try { framework.start(); - await waitFor(() => runsWithError() >= 3); + await waitFor(() => passes >= 3); assert.deepEqual(alerts, ['context-maintenance-failed:fenced']); assert.equal(failedLines(), 1); + assert.equal(runsWithError(), 1, 'the passes that only repeat the failure stay out of the history'); // Recovered with nothing to do: passes find it ready and skip it. await cm.setStrategy(new ReadyStrategy()); @@ -1179,6 +1224,77 @@ describe('queued context maintenance timer', () => { } }); + it("a conversation agent's tick that reaches its provider under a surgery's hold is withdrawn, as a resident's is", { timeout: 10_000 }, async () => { + const dir = mkdtempSync(join(tmpdir(), 'af-maintenance-conversation-withdrawn-')); + const provider = new CapturingMembrane(); + const conversation = new CallAfterGateStrategy(); + const framework = await AgentFramework.create({ + storePath: join(dir, 'store'), + membrane: provider.asMembrane(), + agents: [{ name: 'agent', model: 'test-model', systemPrompt: 'test', strategy: new ReadyStrategy() }], + modules: [], + syncIntervalMs: 0, + maintenanceIntervalMs: 10, + conversations: { templateAgent: 'agent', strategyFactory: () => conversation }, + }); + const internals = framework as unknown as { + createConversationAgent(name: string, channelId: string): Promise; + reserveStoreForSurgery(verb: string, agentName: string): () => void; + }; + const alerts: string[] = []; + (framework as any).opsAlert = (kind: string, agentName: string) => alerts.push(`${kind}:${agentName}`); + const quiet = console.error; + console.error = () => {}; + try { + await internals.createConversationAgent('agent-conv-1', 'chan-1'); + framework.start(); + await waitFor(() => conversation.entered); + const release = internals.reserveStoreForSurgery('roll back', 'agent'); + try { + conversation.release(); + await waitFor(() => conversation.outcome !== undefined); + assert.equal(conversation.outcome, 'MaintenanceWithdrawnError'); + assert.equal(provider.calls.length, 0, 'the withdrawn call sent nothing'); + } finally { + release(); + } + assert.deepEqual(alerts.filter((alert) => alert.startsWith('context-maintenance-failed')), []); + } finally { + console.error = quiet; + conversation.release(); + await framework.stop(); + rmSync(dir, { recursive: true, force: true }); + } + }); + + it('a pass that only meets a failure already reported stays out of the history', { timeout: 10_000 }, async () => { + const dir = mkdtempSync(join(tmpdir(), 'af-maintenance-repeats-')); + const framework = await AgentFramework.create({ + storePath: join(dir, 'store'), + membrane, + agents: [{ name: 'agent', model: 'test-model', systemPrompt: 'test', strategy: new FailingStrategy() }], + modules: [], + syncIntervalMs: 0, + maintenanceIntervalMs: 10, + }); + (framework as any).opsAlert = () => {}; + const internals = framework as unknown as { maintenanceRunId: number }; + const quiet = console.error; + console.error = () => {}; + try { + framework.start(); + await waitFor(() => internals.maintenanceRunId >= 4); + const history = framework.getContextMaintenanceSnapshot().history; + assert.equal(history.length, 1, 'the first report only, though four passes met the failure'); + assert.equal(history[0]!.id, 1); + assert.ok(history[0]!.agents[0]!.error); + } finally { + console.error = quiet; + await framework.stop(); + rmSync(dir, { recursive: true, force: true }); + } + }); + it('a surgery that takes the store while resume waits for a tick is refused by resume after the wait', { timeout: 10_000 }, async () => { const dir = mkdtempSync(join(tmpdir(), 'af-maintenance-resume-rechecks-')); const strategy = new GatedTickStrategy(); @@ -1242,6 +1358,8 @@ describe('queued context maintenance timer', () => { const quiet = console.error; console.error = () => {}; try { + // A drain kick ticks only while the framework runs. + framework.start(); // A refused compile kicks the drain, whose first tick waits on the gate. internals.noteInferenceExhausted('agent', 'compile refused', false, 'over_budget'); await waitFor(() => strategy.ticks === 1); @@ -1261,6 +1379,43 @@ describe('queued context maintenance timer', () => { } }); + it("stop() waits for an over-budget drain kick's tick in flight, and the kick ticks no more once stop() has begun", { timeout: 10_000 }, async () => { + const dir = mkdtempSync(join(tmpdir(), 'af-maintenance-drain-stop-')); + const strategy = new GatedTickStrategy(); + const framework = await AgentFramework.create({ + storePath: join(dir, 'store'), + membrane, + agents: [{ name: 'agent', model: 'test-model', systemPrompt: 'test', strategy }], + modules: [], + syncIntervalMs: 0, + maintenanceIntervalMs: 0, + }); + strategy.framework = framework; + const internals = framework as unknown as { + noteInferenceExhausted(agentName: string, reason: string, retryable?: boolean, errorType?: string): void; + }; + const quiet = console.error; + console.error = () => {}; + let stopped = false; + try { + framework.start(); + // A refused compile kicks the drain, whose first tick waits on the gate. + internals.noteInferenceExhausted('agent', 'compile refused', false, 'over_budget'); + await waitFor(() => strategy.ticks === 1); + const stopping = framework.stop().then(() => { stopped = true; }); + await new Promise((resolve) => setTimeout(resolve, 100)); + assert.equal(stopped, false, 'stop() waits for the kick\'s tick in flight'); + strategy.release(); + await stopping; + assert.equal(strategy.ticks, 1, 'the kick starts no tick once stop() has begun'); + } finally { + console.error = quiet; + strategy.release(); + if (!stopped) await framework.stop(); + rmSync(dir, { recursive: true, force: true }); + } + }); + it('no pass starts while a surgery holds the store', async () => { const dir = mkdtempSync(join(tmpdir(), 'af-maintenance-surgery-hold-')); const strategy = new QueuedStrategy(); diff --git a/test/operator-surgery.test.ts b/test/operator-surgery.test.ts index 46bacc245..9ad180841 100644 --- a/test/operator-surgery.test.ts +++ b/test/operator-surgery.test.ts @@ -429,6 +429,32 @@ describe('live operator surgery', () => { assert.deepEqual(entry.result?.removedIds, huge.slice(0, 50)); assert.equal(entry.result?.removedIdsTruncated, true); for (const id of huge) assert.equal(cm().getMessage(id as never), null); + + // The range form runs from the earlier message's first copy through the + // later one's last, in either order: a range ending at a message stored + // twice takes its retry too. + const hideRange = (fromMessageId: string, toMessageId: string) => internals.handleHostCommand('discord', { + command: 'hide', agentName: 'scout', fromMessageId, toMessageId, requesterName: 'antra', + }); + const plain = (text: string, discordId: string) => cm().addMessage('user', [{ type: 'text', text }], { + serverId: 'discord', channelId: 'discord:g1:c1', messageId: discordId, + }); + appendBody('d-cut2', 'g-cut2', ['p0', 'p1']); + appendBody('d-cut2', 'g-cut2', ['r0', 'r1', 'r2']); + plain('after', 'd-after'); + const forward = await hideRange('d4', 'd-cut2'); + assert.equal(forward.ok, true, String(forward.error)); + assert.equal(forward.hidden, 4, 'm4, m5, and both copies of the cut message'); + assert.deepEqual(refsOf(forward), ['d4', 'd5', 'd-cut2']); + assert.deepEqual(remainingTexts(), ['m1', 'm2', 'm3', 'after']); + + appendBody('d-cut3', 'g-cut3', ['q0']); + appendBody('d-cut3', 'g-cut3', ['s0', 's1']); + plain('last', 'd-last'); + const backward = await hideRange('d-cut3', 'd2'); + assert.equal(backward.ok, true, String(backward.error)); + assert.equal(backward.hidden, 5, 'm2, m3, after, and both copies of the cut message'); + assert.deepEqual(remainingTexts(), ['m1', 'last']); }); it('hide refuses while a surgery holds the store, and a missing Discord id matches nothing', async () => { diff --git a/test/overbudget-drain-breaker.test.ts b/test/overbudget-drain-breaker.test.ts index 3defc651e..f18a589a6 100644 --- a/test/overbudget-drain-breaker.test.ts +++ b/test/overbudget-drain-breaker.test.ts @@ -33,6 +33,8 @@ function makeHarness(tick: () => Promise) { fw.lastInferenceAt = new Map(); fw.overBudgetDrainInFlight = new Set(); fw.overBudgetDrains = new Set(); + // A drain kick ticks only while the framework runs. + fw.running = true; fw.opsAlertLastSent = new Map(); fw.pendingRequests = []; fw.traceListeners = []; From 78af9d877260d788d70a08fff307803eb0f3172b Mon Sep 17 00:00:00 2001 From: Gladys-2413 Date: Thu, 15 Oct 2026 10:02:31 -0400 Subject: [PATCH 02/11] fix: the internal review's round on #352's follow-up Our internal review of the follow-up found these, each fixed here. A hide range missed a copy of its earlier message stored after its later one, and it took what was stored between the later message's copies, which came after that message began. A range now takes what was stored after its earlier message began and before its later one began, from the earlier one's first copy through the later one's first, and every copy of both messages, each whole, as the single form already takes all of a message's copies. Offline recovery's suppressed ranges take the same; before, they ran through the later message's last copy. A pass whose records all met a failure already reported was kept out of the history even when a record had taken a tick first. A pass now stays out only if every record repeats a reported failure and took no tick. No over-budget drain kick starts once stop() has begun, and a conversation agent's compression call made then is refused, as a resident's is. The budget preflight no longer logs that it is applying an update that a refusing manager then refuses. Text: the preview's contract says a patch whose values the agent refuses throws whatever the manager's state, while a patch that sets tail or pace gets no verdict from a refusing manager, since whether the strategy takes them is the manager's to say. reportMaintenanceFailure's comment, the conversation membrane's, contextManagerAnswers' and stop()'s teardown budget now say what the code does. The changelog says when a lasting failure is recorded, and gives the provider gate clause to a resident's call only. Tests: hide's range in both shapes in operator-surgery, and offline recovery's in offline-recovery-branch; in maintenance-timer, a pass that took a tick before the repeated failure stays in the history, a conversation agent's call once stop() has begun is refused, and no drain kick starts then; the stop() test checks that the tick in flight ended before the store closed; in host-quiesce, a budget update under a refusing manager throws and changes nothing. --- changelog.d/body-removals.fixed.md | 2 +- changelog.d/cm-0.14.changed.md | 4 +- src/agent.ts | 8 +- src/framework.ts | 118 ++++++++++++++---------- src/recovery/offline-branch.ts | 17 ++-- test/host-quiesce.test.ts | 16 +++- test/maintenance-timer.test.ts | 131 ++++++++++++++++++++++++++- test/offline-recovery-branch.test.ts | 51 +++++++++++ test/operator-surgery.test.ts | 31 ++++++- 9 files changed, 310 insertions(+), 68 deletions(-) diff --git a/changelog.d/body-removals.fixed.md b/changelog.d/body-removals.fixed.md index a0db3c730..4536a69b3 100644 --- a/changelog.d/body-removals.fixed.md +++ b/changelog.d/body-removals.fixed.md @@ -1 +1 @@ -- Removing a single message that belongs to a body, a one-shard body's included, no longer fails: context-manager's `removeMessage` refuses any message in a body, so agent-framework now removes one as a range in offline recovery's suppressions, and in a Discord suppression resumed after a crash. The host `hide` command now hides a message stored as a body (a long one, in shards) whole, with every stored copy of it (a write cut short leaves a partial copy before its retry). Its range form runs from the earlier message's first copy through the later message's last, as offline recovery's range does. Either form's operator-log entry lists at most 50 of the ids it removed, with `removedIdsTruncated` when there were more. It's refused as busy while a surgery holds the store, and a message with no Discord id never matches one. A module's removal of a message it added, and the coalescer's, take its whole body too, and an automatic rewind whose newest message is a shard of a body takes the whole body. +- Removing a single message that belongs to a body, a one-shard body's included, no longer fails: context-manager's `removeMessage` refuses any message in a body, so agent-framework now removes one as a range in offline recovery's suppressions, and in a Discord suppression resumed after a crash. The host `hide` command now hides a message stored as a body (a long one, in shards) whole, with every stored copy of it (a write cut short leaves a partial copy before its retry). Its range form takes what was stored between its two messages, from the earlier one's first copy through the later one's first, and every copy of both, each whole, so a copy stored outside that span goes alone and nothing stored after the later message began goes with it. Offline recovery's suppressed ranges (`suppressRanges`) take the same: before, a range ran through the later message's last copy, taking what was stored between that message's copies, and left out a copy of the earlier message stored after it. Either form's operator-log entry lists at most 50 of the ids it removed, with `removedIdsTruncated` when there were more. It's refused as busy while a surgery holds the store, and a message with no Discord id never matches one. A module's removal of a message it added, and the coalescer's, take its whole body too, and an automatic rewind whose newest message is a shard of a body takes the whole body. diff --git a/changelog.d/cm-0.14.changed.md b/changelog.d/cm-0.14.changed.md index 7ac6469b1..f712ce11e 100644 --- a/changelog.d/cm-0.14.changed.md +++ b/changelog.d/cm-0.14.changed.md @@ -1,2 +1,2 @@ -- A context-maintenance pass no longer fails as a whole when one agent's readiness can't be read. context-manager 0.14, which agent-framework has required since 0.21.1, refuses its readiness queries while a strategy's initialization is pending (inside `switchBranch`, `fork`, `setStrategy` or `ContextManager.open`) or has failed, and a strategy can refuse on its own, as the autobiographical one does for a branch it hasn't reloaded. That agent's run now records the error, the agent gets its own `context-maintenance-failed` alert (`scope: 'agent'`), and the other agents run. Before, such a throw ended the pass for every agent, with one alert for the whole pass (`scope: 'pass'`). Anything else that throws while one agent's maintenance is prepared is that agent's failure in the same way. A failure that lasts, met by every pass, is recorded in the maintenance history once, and logged and alerted once, then again when its text changes, every 15 minutes while it lasts (the alert webhook's cooldown for it, counted from the webhook's last post of it), or after that agent's maintenance has gone without a failure; a pass that only met failures already reported isn't added to the history, so a lasting one no longer pushes out the runs that did work. `getContextMaintenanceSnapshot()` reports such an agent as `ready: false` with an `error`, and a strategy that can't give a progress snapshot reads as having none. -- While an operator surgery (`rollbackToMessage` or `suppressMessages`) or `switchBranch` holds the store, no maintenance pass starts and no over-budget drain kick is started, a running pass or drain kick starts no new tick, and the operator call waits for them to finish before it switches or forks the branch, so no maintenance work overlaps the move. `stop()` waits for them the same way, a drain kick included, before it closes the store, and a drain kick starts no new tick once `stop()` has begun. Maintenance makes no provider call meanwhile, a conversation agent's included: one made then, or one waiting at the agent's provider gate when the hold is taken, is withdrawn without being sent. That's no maintenance failure, and context-manager counts no attempt for it. A merge keeps its place in the queue. A chunk's compression logs context-manager's "Failed to compress chunk (hierarchical)" line with the withdrawal, and the chunk is queued again at that manager's next rebuild, such as its next message or compile. The operator call never waits on a provider cooldown, or on the retry turn a cooldown queues, which the hold keeps from starting. `switchBranch` can therefore move the store after the call returns, and the API server's `branch.switch` announces the move it made once it has, while `branch.current`, the branch list, `/health` and `/status` read the store's branch whenever they report one. `maintenanceTick()` answers `ran: false` meanwhile, and `resume()` refuses with `OperatorActionError` (`agent-busy`), as the other operator calls do, including when the surgery took the store while `resume()` waited. Resume once the surgery has finished. +- A context-maintenance pass no longer fails as a whole when one agent's readiness can't be read. context-manager 0.14, which agent-framework has required since 0.21.1, refuses its readiness queries while a strategy's initialization is pending (inside `switchBranch`, `fork`, `setStrategy` or `ContextManager.open`) or has failed, and a strategy can refuse on its own, as the autobiographical one does for a branch it hasn't reloaded. That agent's run now records the error, the agent gets its own `context-maintenance-failed` alert (`scope: 'agent'`), and the other agents run. Before, such a throw ended the pass for every agent, with one alert for the whole pass (`scope: 'pass'`). Anything else that throws while one agent's maintenance is prepared is that agent's failure in the same way. A failure that lasts, met by every pass, is logged, alerted and recorded in the maintenance history when it's first met, then again when its text changes, every 15 minutes while it lasts (the alert webhook's cooldown for it, counted from the webhook's last post of it), or after that agent's maintenance has gone without a failure. A pass that met only failures already reported, and took no tick, isn't added to the history, so a lasting failure no longer pushes out the runs that did work. `getContextMaintenanceSnapshot()` reports such an agent as `ready: false` with an `error`, and a strategy that can't give a progress snapshot reads as having none. +- While an operator surgery (`rollbackToMessage` or `suppressMessages`) or `switchBranch` holds the store, no maintenance pass starts and no over-budget drain kick is started, a running pass or drain kick starts no new tick, and the operator call waits for them to finish before it switches or forks the branch, so no maintenance work overlaps the move. `stop()` waits for them the same way, a drain kick included, before it closes the store, and once `stop()` has begun, no drain kick starts, none starts a new tick, and a conversation agent's compression calls are refused, as a resident's are. Maintenance makes no provider call meanwhile, a conversation agent's included: one made then is withdrawn without being sent, and so is a resident's call waiting at its provider gate when the hold is taken. That's no maintenance failure, and context-manager counts no attempt for it. A merge keeps its place in the queue. A chunk's compression logs context-manager's "Failed to compress chunk (hierarchical)" line with the withdrawal, and the chunk is queued again at that manager's next rebuild, such as its next message or compile. The operator call never waits on a provider cooldown, or on the retry turn a cooldown queues, which the hold keeps from starting. `switchBranch` can therefore move the store after the call returns, and the API server's `branch.switch` announces the move it made once it has, while `branch.current`, the branch list, `/health` and `/status` read the store's branch whenever they report one. `maintenanceTick()` answers `ran: false` meanwhile, and `resume()` refuses with `OperatorActionError` (`agent-busy`), as the other operator calls do, including when the surgery took the store while `resume()` waited. Resume once the surgery has finished. diff --git a/src/agent.ts b/src/agent.ts index 117fda103..01d26752a 100644 --- a/src/agent.ts +++ b/src/agent.ts @@ -517,9 +517,11 @@ export class Agent { } /** - * Whether the context manager answers its queries. It refuses them while - * its strategy's initialization is pending or has failed, until a - * switchBranch or setStrategy succeeds. + * Whether the context manager answers its queries, as its readiness says. + * It refuses them while its strategy's initialization is pending or has + * failed, until a switchBranch or setStrategy succeeds, and a strategy can + * refuse its readiness on its own, as the autobiographical one does for a + * branch it hasn't reloaded. */ contextManagerAnswers(): boolean { try { diff --git a/src/framework.ts b/src/framework.ts index b2325f51f..b96d41eaa 100644 --- a/src/framework.ts +++ b/src/framework.ts @@ -2543,7 +2543,8 @@ export class AgentFramework { * @param teardownTimeoutMs Optional budget for each wait on cleanup: an * operator's operations already running, then modules and MCPL together. * Omit to wait until cleanup settles, as required by existing module/child - * contracts. This does not bound earlier stream or event-loop quiescence, or + * contracts. This does not bound earlier stream, event-loop or maintenance + * quiescence (a maintenance pass or over-budget drain kick in flight), or * cancel cleanup. */ stop(teardownTimeoutMs = DEFAULT_SHUTDOWN_TIMEOUT_MS): Promise { @@ -2898,14 +2899,19 @@ export class AgentFramework { * as a resident's are, so they bind its auxiliary calls too: a held model * is refused without a call, and an auxiliary call's own stated wait is * recorded for it. (An ephemeral run records nothing, its name being - * single-use, so its context manager keeps the plain membrane.) A - * conversation agent is ticked by the maintenance pass too, so a call made - * while a surgery holds the store is withdrawn, as a resident's is. + * single-use, so its context manager keeps the plain membrane.) Its + * compression is the background work a resident's is, whether a + * maintenance tick or a drain after a message starts it, so it meets the + * same refusals: a call made while a surgery holds the store is withdrawn, + * and one made once stop() has closed provider admission is refused. */ private conversationAuxiliaryMembraneFor(agentName: string): Membrane { return this.membraneAdmitting(async (run, model) => { this.refuseHeldProviderModel(agentName, model); this.withdrawMaintenanceUnderSurgery(); + if (this.providerAdmissionClosed) { + throw new Error(`Provider admission closed while stopping (${agentName})`); + } try { return await run(); } catch (error) { if (model !== undefined && error instanceof Error) this.recordProviderWait(agentName, model, error); @@ -3537,10 +3543,10 @@ export class AgentFramework { run.finishedAt = Date.now(); run.agents.sort((a, b) => a.agentName.localeCompare(b.agentName)); this.currentMaintenanceRun = null; - // A pass that only met failures already reported adds nothing the - // history doesn't hold, and a lasting one would otherwise fill it in - // minutes, pushing out the runs that did work. - if (!run.agents.every((record) => this.maintenanceRepeats.has(record))) this.maintenanceHistory.unshift(run); + // A pass that only met failures already reported, and took no tick, + // adds nothing the history doesn't hold, and a lasting failure would + // otherwise fill it in minutes, pushing out the runs that did work. + if (!run.agents.every((record) => record.ticks === 0 && this.maintenanceRepeats.has(record))) this.maintenanceHistory.unshift(run); if (this.maintenanceHistory.length > 50) this.maintenanceHistory.length = 50; } } @@ -3549,12 +3555,14 @@ export class AgentFramework { * One agent's maintenance failure: recorded on its run, bounded as every * failure sink is (`record.error`), and logged and alerted for that agent * with the whole text. A failure that lasts, such as a manager whose - * strategy failed to initialize, meets every pass, a few seconds apart: - * each run records it, and it is logged and alerted again only when its - * whole text changes, once opsAlert's cooldown for it has passed since the - * webhook last posted it (so the webhook, which posts a kind at most once - * per cooldown, hears a failure that lasts), or after a pass in which that - * agent's maintenance went without a failure (maintenanceFailureReported). + * strategy failed to initialize, meets every pass, a few seconds apart. It + * is logged and alerted again only when its whole text changes, once + * opsAlert's cooldown for it has passed since the webhook last posted it + * (so the webhook, which posts a kind at most once per cooldown, hears a + * failure that lasts), or after a pass in which that agent's maintenance + * went without a failure (maintenanceFailureReported). A record whose + * failure isn't reported again is marked as a repeat, and a pass whose + * every record is a repeat that took no tick stays out of the history. */ private reportMaintenanceFailure(agentName: string, record: ContextMaintenanceAgentRun, wholeError: string): void { record.error = boundFailureText(wholeError); @@ -4366,7 +4374,8 @@ export class AgentFramework { `advance merges to go lower). Applying; the live compile budget is unchanged.`, ); } - } else if (preview.reason) { + } else if (preview.reason && agent.contextManagerAnswers()) { + // A manager that refuses its queries refuses the update below too. console.warn( `[budget-preflight] preview unavailable for ${agentName} (${preview.reason}); ` + `applying without preflight`, @@ -5300,7 +5309,10 @@ export class AgentFramework { * adaptive) path, and a context manager that refuses its queries (its * strategy's initialization pending or failed) all come back as * `{available: false, reason}`. Unknown agent still throws, matching the - * sibling accessors, and so does a patch the agent refuses. + * sibling accessors, and so does a patch whose values the agent refuses, + * whatever its context manager's state. Whether the strategy takes live + * tail and pace settings is the manager's to say, so while it refuses, a + * patch that sets them gets no verdict either. */ previewAgentRuntimeSettings( agentName: string, @@ -5316,7 +5328,8 @@ export class AgentFramework { agent: Agent, patch?: AgentRuntimeSettingsPatch, ): RuntimeSettingsPreview { - // A patch the agent refuses throws, whatever its context manager's state. + // A patch whose values the agent refuses throws, whatever its context + // manager's state. if (patch && Object.keys(patch).length > 0) agent.checkRuntimeSettingsPatch(patch); let plan: ReturnType; let settings: ReturnType; @@ -6998,12 +7011,26 @@ export class AgentFramework { return id === undefined || id === null || id === '' ? undefined : String(id); }; const byDiscordId = (did: string) => all.findIndex((m) => discordIdOf(m) === String(did)); - const lastByDiscordId = (did: string) => { - for (let i = all.length - 1; i >= 0; i--) if (discordIdOf(all[i]) === String(did)) return i; - return -1; + /** Runs in stored order, those that overlap or touch made one. */ + const merged = (runs: Array<{ from: number; to: number }>) => { + const out: Array<{ from: number; to: number }> = []; + for (const { from, to } of [...runs].sort((a, b) => a.from - b.from)) { + const last = out[out.length - 1]; + if (last && from <= last.to + 1) last.to = Math.max(last.to, to); + else out.push({ from, to }); + } + return out; + }; + // A message stored as a body (a long one, in shards) goes whole, and so + // does every copy of it: a write cut short leaves a partial body before + // its retry, both carrying the Discord id. + const copiesOf = (did: string) => { + const runs: Array<{ from: number; to: number }> = []; + all.forEach((m, i) => { if (discordIdOf(m) === String(did)) runs.push(bodyRangeIn(all, i)); }); + return runs; }; - // Collect Discord refs of every message in [lo, hi] that carries one, + // Collect Discord refs of every message in the runs that carries one, // so the surface can mark them. channelId here is whatever the ingest // path stored (raw id or "discord:guild:channel"); the surface // normalizes it. @@ -7047,20 +7074,22 @@ export class AgentFramework { if (toIdx < 0) { return { ok: false, error: `Message ${params.toMessageId} is not an addressable message in context.` }; } - // The earlier end from its first stored copy, the later end through - // its last, as offline recovery's range does: a write cut short and - // retried stores a message twice, and the range takes both copies. - const [first, laterId] = fromIdx <= toIdx - ? [fromIdx, params.toMessageId] - : [toIdx, params.fromMessageId]; - const last = lastByDiscordId(laterId); - // A message stored as a body (a long one, in shards) is hidden whole. - const lo = bodyRangeIn(all, first).from; - const hi = bodyRangeIn(all, last).to; - const refs = refsIn([{ from: lo, to: hi }]); - const hidden = messagesIn([{ from: lo, to: hi }]); - const rangeIds = all.slice(lo, hi + 1).map((m) => String(m.id)); - removeStoredMessages(cm, all[lo], all[hi]); + // What's between the two messages, after the earlier one began and + // before the later one began: from the earlier one's first stored + // copy through the later one's first, each whole. Each end is every + // copy of it besides, as the single form takes, so a copy stored + // outside that span (a write cut short and retried later) goes + // alone, and nothing stored after the later one began goes with it. + // Offline recovery's range is the same. + const span = { + from: bodyRangeIn(all, Math.min(fromIdx, toIdx)).from, + to: bodyRangeIn(all, Math.max(fromIdx, toIdx)).to, + }; + const runs = merged([span, ...copiesOf(params.fromMessageId), ...copiesOf(params.toMessageId)]); + const refs = refsIn(runs); + const hidden = messagesIn(runs); + const rangeIds = runs.flatMap(({ from, to }) => all.slice(from, to + 1).map((m) => String(m.id))); + for (const { from, to } of runs) removeStoredMessages(cm, all[from], all[to]); console.error( `[host-command] hide agent=${agentName} range removed=${hidden} ` + `(${params.fromMessageId}..${params.toMessageId}) by=${params.requesterName ?? params.requesterId ?? 'unknown'} (server=${serverId})`, @@ -7085,17 +7114,7 @@ export class AgentFramework { lastVisible: await this.lastVisiblePreview(agentName), }; } - // A message stored as a body (a long one, in shards) is hidden whole, - // and so is every copy of it: a write cut short leaves a partial body - // before its retry, both carrying the Discord id. - const runs: Array<{ from: number; to: number }> = []; - all.forEach((m, i) => { - if (discordIdOf(m) !== String(params.fromMessageId)) return; - const { from, to } = bodyRangeIn(all, i); - const last = runs[runs.length - 1]; - if (last && from <= last.to + 1) last.to = Math.max(last.to, to); - else runs.push({ from, to }); - }); + const runs = merged(copiesOf(params.fromMessageId)); const refs = refsIn(runs); const hidden = messagesIn(runs); const hiddenIds = runs.flatMap(({ from, to }) => all.slice(from, to + 1).map((m) => String(m.id))); @@ -15750,10 +15769,11 @@ export class AgentFramework { { skipLog: true }, ); } - if (agent && overBudget && !this.overBudgetDrainInFlight.has(agentName) && !this.surgeryHold) { + if (agent && overBudget && this.running && !this.overBudgetDrainInFlight.has(agentName) && !this.surgeryHold) { this.overBudgetDrainInFlight.add(agentName); - // Like a maintenance pass: it starts no tick while a surgery holds the - // store, and a branch move waits for the tick in flight. + // Like a maintenance pass: none starts once stop() has begun, it starts + // no tick while a surgery holds the store, and a branch move, or stop(), + // waits for the tick in flight. const drain = (async () => { let ticks = 0; try { diff --git a/src/recovery/offline-branch.ts b/src/recovery/offline-branch.ts index 49137f1af..1bd3848b3 100644 --- a/src/recovery/offline-branch.ts +++ b/src/recovery/offline-branch.ts @@ -240,9 +240,7 @@ interface SuppressionInterval { interface MessageLocation { first: number; - last: number; firstId: StoredWindowMessage['id']; - lastId: StoredWindowMessage['id']; /** Usually one run; multiple runs tolerate duplicate external metadata. */ runs: SuppressionInterval[]; } @@ -405,9 +403,7 @@ function buildSuppressionPlan( }; locations.set(externalId, { first: index, - last: index, firstId: messages[i].id, - lastId: messages[i].id, runs: [run], }); } else { @@ -423,8 +419,6 @@ function buildSuppressionPlan( toId: messages[i].id, }); } - location.last = index; - location.lastId = messages[i].id; } } } @@ -447,12 +441,19 @@ function buildSuppressionPlan( const to = locations.get(range.toMessageId)!; const earlier = from.first <= to.first ? from : to; const later = from.first <= to.first ? to : from; + // What's between the two messages, after the earlier one began and + // before the later one began: from the earlier one's first stored copy + // through the later one's first. Each end is every copy of it besides, + // as a single suppression is, so a copy stored outside that span (a + // write cut short and retried later) goes alone, and nothing stored + // after the later one began goes with it. intervals.push({ start: earlier.first, - end: later.last, + end: later.first, fromId: earlier.firstId, - toId: later.lastId, + toId: later.firstId, }); + for (const end of [from, to]) intervals.push(...end.runs.map((run) => ({ ...run }))); } intervals.sort((a, b) => a.start - b.start || a.end - b.end); diff --git a/test/host-quiesce.test.ts b/test/host-quiesce.test.ts index c1fe9680d..da7eba3ae 100644 --- a/test/host-quiesce.test.ts +++ b/test/host-quiesce.test.ts @@ -339,9 +339,23 @@ test('resume proceeds without a verdict for an agent whose context manager refus assert.equal(preview.liveBudgetTokens, 100_000); assert.equal(preview.transition, 'stable'); assert.equal(preview.effective, undefined); - // A patch the agent refuses still throws. + // A patch whose values the agent refuses still throws. assert.throws(() => framework.previewAgentRuntimeSettings('agent', { contextBudgetTokens: 5 }), /contextBudgetTokens must be/); + // An update of the budget is refused, by the manager, and the preflight + // doesn't say it's applying one. + const preflight = t.mock.method(console, 'warn', () => {}); + try { + assert.throws( + () => framework.updateAgentRuntimeSettings('agent', { contextBudgetTokens: 50_000 }), + /requires successful strategy initialization/, + ); + assert.ok(!preflight.mock.calls.some((call) => String(call.arguments[0]).includes('applying without preflight'))); + } finally { + preflight.mock.restore(); + } + assert.equal(framework.getAgent('agent')!.runtimeBudgetPath().liveBudgetTokens, 100_000, 'the budget is as it was'); + const warn = t.mock.method(console, 'warn', () => {}); let status: Awaited>; try { diff --git a/test/maintenance-timer.test.ts b/test/maintenance-timer.test.ts index bf6b16139..5bc071cd0 100644 --- a/test/maintenance-timer.test.ts +++ b/test/maintenance-timer.test.ts @@ -101,6 +101,29 @@ class FailingStrategy implements ContextStrategy { } } +/** A strategy whose every pass takes one tick, then meets the same failure. */ +class WorkThenFailStrategy implements ContextStrategy { + readonly name = 'work-then-fail-test'; + ticks = 0; + + checkReadiness(): ReadinessState { + return { ready: false, description: 'compression queued' }; + } + + async tick(_ctx: StrategyContext): Promise { + this.ticks++; + if (this.ticks % 2 === 0) throw new Error('provider rejected maintenance request'); + } + + select( + _store: MessageStoreView, + _log: ContextLogView, + _budget: TokenBudget, + ): ContextEntry[] { + return []; + } +} + /** A strategy whose initialization always fails. */ class RefusingInitStrategy implements ContextStrategy { readonly name = 'refusing-init-test'; @@ -257,6 +280,7 @@ class GatedTickStrategy implements ContextStrategy { ticks = 0; ticksUnderHold = 0; branchAtFirstTickEnd?: string; + storeOpenAtFirstTickEnd?: boolean; framework?: AgentFramework; private open?: () => void; private readonly gate = new Promise((resolve) => { this.open = resolve; }); @@ -274,6 +298,7 @@ class GatedTickStrategy implements ContextStrategy { if ((this.framework as unknown as { surgeryHold: unknown }).surgeryHold) this.ticksUnderHold++; if (this.ticks === 1) { await this.gate; + this.storeOpenAtFirstTickEnd = !(this.framework as unknown as { store: { isClosed(): boolean } }).store.isClosed(); this.branchAtFirstTickEnd = this.framework!.getAgent('agent')!.getContextManager().currentBranch().name; } } @@ -335,6 +360,7 @@ class CallAfterGateStrategy implements ContextStrategy { ticks = 0; entered = false; outcome?: string; + outcomeMessage?: string; private open?: () => void; private readonly gate = new Promise((resolve) => { this.open = resolve; }); @@ -359,6 +385,7 @@ class CallAfterGateStrategy implements ContextStrategy { this.outcome = 'sent'; } catch (error) { this.outcome = (error as Error).name; + this.outcomeMessage = (error as Error).message; throw error; } } @@ -1295,6 +1322,105 @@ describe('queued context maintenance timer', () => { } }); + it('a pass that took a tick before meeting a failure already reported stays in the history', { timeout: 10_000 }, async () => { + const dir = mkdtempSync(join(tmpdir(), 'af-maintenance-repeats-after-work-')); + const framework = await AgentFramework.create({ + storePath: join(dir, 'store'), + membrane, + agents: [{ name: 'agent', model: 'test-model', systemPrompt: 'test', strategy: new WorkThenFailStrategy() }], + modules: [], + syncIntervalMs: 0, + maintenanceIntervalMs: 10, + }); + (framework as any).opsAlert = () => {}; + const internals = framework as unknown as { maintenanceRunId: number }; + const quiet = console.error; + console.error = () => {}; + try { + framework.start(); + await waitFor(() => internals.maintenanceRunId >= 4); + const history = framework.getContextMaintenanceSnapshot().history; + assert.ok(history.length >= 3, `each finished pass did work, so each is kept (${history.length})`); + for (const run of history) { + assert.equal(run.agents[0]!.ticks, 1); + assert.ok(run.agents[0]!.error); + } + } finally { + console.error = quiet; + await framework.stop(); + rmSync(dir, { recursive: true, force: true }); + } + }); + + it("a conversation agent's call made once stop() has begun is refused, as a resident's is", { timeout: 10_000 }, async () => { + const dir = mkdtempSync(join(tmpdir(), 'af-maintenance-conversation-stopping-')); + const provider = new CapturingMembrane(); + const conversation = new CallAfterGateStrategy(); + const framework = await AgentFramework.create({ + storePath: join(dir, 'store'), + membrane: provider.asMembrane(), + agents: [{ name: 'agent', model: 'test-model', systemPrompt: 'test', strategy: new ReadyStrategy() }], + modules: [], + syncIntervalMs: 0, + maintenanceIntervalMs: 10, + conversations: { templateAgent: 'agent', strategyFactory: () => conversation }, + }); + const internals = framework as unknown as { + createConversationAgent(name: string, channelId: string): Promise; + }; + (framework as any).opsAlert = () => {}; + const quiet = console.error; + console.error = () => {}; + let stopping: Promise | undefined; + try { + await internals.createConversationAgent('agent-conv-1', 'chan-1'); + framework.start(); + await waitFor(() => conversation.entered); + stopping = framework.stop(); + conversation.release(); + await waitFor(() => conversation.outcome !== undefined); + assert.match(conversation.outcomeMessage ?? '', /Provider admission closed while stopping \(agent-conv-1\)/); + assert.equal(provider.calls.length, 0, 'the refused call sent nothing'); + await stopping; + } finally { + console.error = quiet; + conversation.release(); + await (stopping ?? framework.stop()); + rmSync(dir, { recursive: true, force: true }); + } + }); + + it('no over-budget drain kick starts once stop() has begun', async () => { + const dir = mkdtempSync(join(tmpdir(), 'af-maintenance-drain-after-stop-')); + const framework = await AgentFramework.create({ + storePath: join(dir, 'store'), + membrane, + agents: [{ name: 'agent', model: 'test-model', systemPrompt: 'test', strategy: new ReadyStrategy() }], + modules: [], + syncIntervalMs: 0, + maintenanceIntervalMs: 0, + }); + const internals = framework as unknown as { + noteInferenceExhausted(agentName: string, reason: string, retryable?: boolean, errorType?: string): void; + overBudgetDrains: Set>; + }; + (framework as any).opsAlert = () => {}; + const quiet = console.error; + const logged: string[] = []; + console.error = (...args: unknown[]) => { logged.push(args.map(String).join(' ')); }; + try { + framework.start(); + const stopping = framework.stop(); + internals.noteInferenceExhausted('agent', 'compile refused', false, 'over_budget'); + assert.equal(internals.overBudgetDrains.size, 0, 'no kick was started'); + await stopping; + assert.ok(!logged.some((line) => line.includes('drain kicked')), logged.join('\n')); + } finally { + console.error = quiet; + rmSync(dir, { recursive: true, force: true }); + } + }); + it('a surgery that takes the store while resume waits for a tick is refused by resume after the wait', { timeout: 10_000 }, async () => { const dir = mkdtempSync(join(tmpdir(), 'af-maintenance-resume-rechecks-')); const strategy = new GatedTickStrategy(); @@ -1403,10 +1529,13 @@ describe('queued context maintenance timer', () => { internals.noteInferenceExhausted('agent', 'compile refused', false, 'over_budget'); await waitFor(() => strategy.ticks === 1); const stopping = framework.stop().then(() => { stopped = true; }); - await new Promise((resolve) => setTimeout(resolve, 100)); + // A stop() that didn't wait would close the store with the tick held: + // it gets half a second to. + await waitFor(() => stopped, 500).catch(() => {}); assert.equal(stopped, false, 'stop() waits for the kick\'s tick in flight'); strategy.release(); await stopping; + assert.equal(strategy.storeOpenAtFirstTickEnd, true, 'the tick in flight ended before the store closed'); assert.equal(strategy.ticks, 1, 'the kick starts no tick once stop() has begun'); } finally { console.error = quiet; diff --git a/test/offline-recovery-branch.test.ts b/test/offline-recovery-branch.test.ts index 640cc0f59..bd2a61a6b 100644 --- a/test/offline-recovery-branch.test.ts +++ b/test/offline-recovery-branch.test.ts @@ -190,6 +190,57 @@ test('offline recovery suppresses inclusive ranges in context order', async () = } }); +test('a suppressed range takes every copy of both its messages, and leaves what was stored after the later one began', async () => { + const dir = mkdtempSync(join(tmpdir(), 'offline-recovery-range-copies-')); + const storePath = join(dir, 'agent.chronicle'); + try { + const store = JsStore.openOrCreate({ path: storePath }); + const cm = await ContextManager.open({ store, namespace: 'agents/cairn' }); + const add = (text: string, messageId: string) => cm.addMessage('user', [{ type: 'text', text }], { + serverId: 'discord', channelId: 'discord:g1:c1', messageId, + }); + add('keep', 'm-keep'); + // B was stored twice, a write cut short and retried, with Z between: + // Z came after B began, so it isn't between A and B. + add('A', 'm-a'); + add('B, cut short', 'm-b'); + add('Z', 'm-z'); + add('B, retried', 'm-b'); + // C's retry was stored after D and E: it goes too, on its own. + add('C, cut short', 'm-c'); + add('D', 'm-d'); + add('E', 'm-e'); + add('C, retried', 'm-c'); + add('current anchor', 'm-current'); + cm.close(); + store.close(); + + const result = await createOfflineRecoveryBranch({ + storePath, + agentName: 'cairn', + messageId: 'm-current', + suppressRanges: [ + { fromMessageId: 'm-a', toMessageId: 'm-b' }, + { fromMessageId: 'm-d', toMessageId: 'm-c' }, + ], + branchName: 'recovery/cairn/range-copies', + }); + + assert.equal(result.messagesSuppressed, 6); + assert.deepEqual(result.refs.map((ref) => ref.messageId), ['m-a', 'm-b', 'm-c', 'm-d']); + const recoveredStore = JsStore.openOrCreate({ path: storePath }); + const recoveredCm = await ContextManager.open({ store: recoveredStore, namespace: 'agents/cairn' }); + assert.deepEqual( + recoveredCm.getAllMessages().map((message) => message.metadata?.messageId), + ['m-keep', 'm-z', 'm-e', 'm-current'], + ); + recoveredCm.close(); + recoveredStore.close(); + } finally { + rmSync(dir, { recursive: true, force: true }); + } +}); + test('offline recovery accepts an exact internal context ID anchor', async () => { const dir = mkdtempSync(join(tmpdir(), 'offline-recovery-context-id-')); const storePath = join(dir, 'agent.chronicle'); diff --git a/test/operator-surgery.test.ts b/test/operator-surgery.test.ts index 9ad180841..f56ba01ad 100644 --- a/test/operator-surgery.test.ts +++ b/test/operator-surgery.test.ts @@ -430,9 +430,10 @@ describe('live operator surgery', () => { assert.equal(entry.result?.removedIdsTruncated, true); for (const id of huge) assert.equal(cm().getMessage(id as never), null); - // The range form runs from the earlier message's first copy through the - // later one's last, in either order: a range ending at a message stored - // twice takes its retry too. + // The range form takes what was stored between its two messages, from the + // earlier one's first copy through the later one's first, in either + // order, and every copy of both: a range ending at a message stored twice + // takes its retry too. const hideRange = (fromMessageId: string, toMessageId: string) => internals.handleHostCommand('discord', { command: 'hide', agentName: 'scout', fromMessageId, toMessageId, requesterName: 'antra', }); @@ -455,6 +456,30 @@ describe('live operator surgery', () => { assert.equal(backward.ok, true, String(backward.error)); assert.equal(backward.hidden, 5, 'm2, m3, after, and both copies of the cut message'); assert.deepEqual(remainingTexts(), ['m1', 'last']); + + // What was stored between a message's first copy and its retry came after + // that message began, so a range ending at it leaves that in. + plain('A', 'd-a'); + appendBody('d-b', 'g-b', ['bp0']); + plain('Z', 'd-z'); + appendBody('d-b', 'g-b', ['br0', 'br1']); + const between = await hideRange('d-a', 'd-b'); + assert.equal(between.ok, true, String(between.error)); + assert.equal(between.hidden, 3, 'A, and both copies of B'); + assert.deepEqual(refsOf(between), ['d-a', 'd-b']); + assert.deepEqual(remainingTexts(), ['m1', 'last', 'Z']); + + // A copy of the earlier message stored after the later one goes too, on + // its own: what was stored between them stays. + appendBody('d-c', 'g-c', ['cp0']); + plain('D', 'd-d'); + plain('E', 'd-e'); + appendBody('d-c', 'g-c', ['cr0', 'cr1']); + const mirror = await hideRange('d-d', 'd-c'); + assert.equal(mirror.ok, true, String(mirror.error)); + assert.equal(mirror.hidden, 3, 'both copies of C, and D'); + assert.deepEqual(refsOf(mirror), ['d-c', 'd-d']); + assert.deepEqual(remainingTexts(), ['m1', 'last', 'Z', 'E']); }); it('hide refuses while a surgery holds the store, and a missing Discord id matches nothing', async () => { From a4908918a13aa43d328321ee4d1003055e7f8d4d Mon Sep 17 00:00:00 2001 From: Gladys-2413 Date: Thu, 15 Oct 2026 17:24:07 -0400 Subject: [PATCH 03/11] fix(maintenance): a stop withdraws a maintenance call, and the review's text points Our internal review's second round on the follow-up found these, each fixed here. A maintenance provider call made once stop() had begun was refused with a plain Error, so the maintenance pass reported it as that agent's failure, logged and alerted as context-maintenance-failed and recorded in the history, and a drain kick logged it as a failed kick. A restart that caught a tick before one of its calls paged whoever reads the alerts. Residents have had this since stop() closed provider admission for auxiliary calls; the follow-up's parity gave it to conversation agents too. The call is now withdrawn, as under a surgery's hold: both membranes throw MaintenanceWithdrawnError, whose message names the agent ("provider admission closed while stopping (name)"), so the pass logs that the agent's maintenance stopped and reports no failure, and a drain kick logs that it stopped. MaintenanceWithdrawnError now takes its reason as text. An over-budget compile's context-refusal alert said "drain breaker kicking" whenever the agent was found, though no kick starts once stop() has begun, while a surgery holds the store, or while one is already draining. It now says which, decided where the kick is. Text: hide's and offline recovery's range comments, and the changelog, say a copy outside the span is taken on its own and nothing else stored after the later message began is taken; the preview's catch says that while the manager refuses, whatever stopped the preview is its reason; the changelog says the tick in flight ends before the store closes. Tests: in maintenance-timer, a resident's and a conversation agent's call made once stop() has begun are withdrawn, each logging that its maintenance stopped, with no alert and no failure in the history, and the stop-time drain test checks the alert's wording; in host-quiesce, a tail or pace patch gets no verdict from a refusing manager, and a refused tail still throws; in offline-recovery-branch, a range whose two messages are stored in shards takes each whole. --- changelog.d/body-removals.fixed.md | 2 +- changelog.d/cm-0.14.changed.md | 2 +- src/framework.ts | 78 ++++++++++++++--------- src/recovery/offline-branch.ts | 4 +- test/host-quiesce.test.ts | 9 +++ test/maintenance-timer.test.ts | 92 ++++++++++++++++------------ test/offline-recovery-branch.test.ts | 41 +++++++++++++ 7 files changed, 155 insertions(+), 73 deletions(-) diff --git a/changelog.d/body-removals.fixed.md b/changelog.d/body-removals.fixed.md index 4536a69b3..1df396d8e 100644 --- a/changelog.d/body-removals.fixed.md +++ b/changelog.d/body-removals.fixed.md @@ -1 +1 @@ -- Removing a single message that belongs to a body, a one-shard body's included, no longer fails: context-manager's `removeMessage` refuses any message in a body, so agent-framework now removes one as a range in offline recovery's suppressions, and in a Discord suppression resumed after a crash. The host `hide` command now hides a message stored as a body (a long one, in shards) whole, with every stored copy of it (a write cut short leaves a partial copy before its retry). Its range form takes what was stored between its two messages, from the earlier one's first copy through the later one's first, and every copy of both, each whole, so a copy stored outside that span goes alone and nothing stored after the later message began goes with it. Offline recovery's suppressed ranges (`suppressRanges`) take the same: before, a range ran through the later message's last copy, taking what was stored between that message's copies, and left out a copy of the earlier message stored after it. Either form's operator-log entry lists at most 50 of the ids it removed, with `removedIdsTruncated` when there were more. It's refused as busy while a surgery holds the store, and a message with no Discord id never matches one. A module's removal of a message it added, and the coalescer's, take its whole body too, and an automatic rewind whose newest message is a shard of a body takes the whole body. +- Removing a single message that belongs to a body, a one-shard body's included, no longer fails: context-manager's `removeMessage` refuses any message in a body, so agent-framework now removes one as a range in offline recovery's suppressions, and in a Discord suppression resumed after a crash. The host `hide` command now hides a message stored as a body (a long one, in shards) whole, with every stored copy of it (a write cut short leaves a partial copy before its retry). Its range form takes what was stored between its two messages, from the earlier one's first copy through the later one's first, and every copy of both, each whole, so a copy stored outside that span is taken on its own, and nothing else stored after the later message began is taken. Offline recovery's suppressed ranges (`suppressRanges`) take the same: before, a range ran through the later message's last copy, taking what was stored between that message's copies, and left out a copy of the earlier message stored after it. Either form's operator-log entry lists at most 50 of the ids it removed, with `removedIdsTruncated` when there were more. It's refused as busy while a surgery holds the store, and a message with no Discord id never matches one. A module's removal of a message it added, and the coalescer's, take its whole body too, and an automatic rewind whose newest message is a shard of a body takes the whole body. diff --git a/changelog.d/cm-0.14.changed.md b/changelog.d/cm-0.14.changed.md index f712ce11e..9abed625c 100644 --- a/changelog.d/cm-0.14.changed.md +++ b/changelog.d/cm-0.14.changed.md @@ -1,2 +1,2 @@ - A context-maintenance pass no longer fails as a whole when one agent's readiness can't be read. context-manager 0.14, which agent-framework has required since 0.21.1, refuses its readiness queries while a strategy's initialization is pending (inside `switchBranch`, `fork`, `setStrategy` or `ContextManager.open`) or has failed, and a strategy can refuse on its own, as the autobiographical one does for a branch it hasn't reloaded. That agent's run now records the error, the agent gets its own `context-maintenance-failed` alert (`scope: 'agent'`), and the other agents run. Before, such a throw ended the pass for every agent, with one alert for the whole pass (`scope: 'pass'`). Anything else that throws while one agent's maintenance is prepared is that agent's failure in the same way. A failure that lasts, met by every pass, is logged, alerted and recorded in the maintenance history when it's first met, then again when its text changes, every 15 minutes while it lasts (the alert webhook's cooldown for it, counted from the webhook's last post of it), or after that agent's maintenance has gone without a failure. A pass that met only failures already reported, and took no tick, isn't added to the history, so a lasting failure no longer pushes out the runs that did work. `getContextMaintenanceSnapshot()` reports such an agent as `ready: false` with an `error`, and a strategy that can't give a progress snapshot reads as having none. -- While an operator surgery (`rollbackToMessage` or `suppressMessages`) or `switchBranch` holds the store, no maintenance pass starts and no over-budget drain kick is started, a running pass or drain kick starts no new tick, and the operator call waits for them to finish before it switches or forks the branch, so no maintenance work overlaps the move. `stop()` waits for them the same way, a drain kick included, before it closes the store, and once `stop()` has begun, no drain kick starts, none starts a new tick, and a conversation agent's compression calls are refused, as a resident's are. Maintenance makes no provider call meanwhile, a conversation agent's included: one made then is withdrawn without being sent, and so is a resident's call waiting at its provider gate when the hold is taken. That's no maintenance failure, and context-manager counts no attempt for it. A merge keeps its place in the queue. A chunk's compression logs context-manager's "Failed to compress chunk (hierarchical)" line with the withdrawal, and the chunk is queued again at that manager's next rebuild, such as its next message or compile. The operator call never waits on a provider cooldown, or on the retry turn a cooldown queues, which the hold keeps from starting. `switchBranch` can therefore move the store after the call returns, and the API server's `branch.switch` announces the move it made once it has, while `branch.current`, the branch list, `/health` and `/status` read the store's branch whenever they report one. `maintenanceTick()` answers `ran: false` meanwhile, and `resume()` refuses with `OperatorActionError` (`agent-busy`), as the other operator calls do, including when the surgery took the store while `resume()` waited. Resume once the surgery has finished. +- While an operator surgery (`rollbackToMessage` or `suppressMessages`) or `switchBranch` holds the store, no maintenance pass starts and no over-budget drain kick is started, a running pass or drain kick starts no new tick, and the operator call waits for them to finish before it switches or forks the branch, so no maintenance work overlaps the move. `stop()` waits for them the same way, a drain kick included, before it closes the store, so the tick in flight ends before the store closes, and once `stop()` has begun, no drain kick starts and none starts a new tick. Maintenance makes no provider call meanwhile, a conversation agent's included: one made under the hold, or once `stop()` has begun, is withdrawn without being sent, and so is a resident's call waiting at its provider gate when the hold is taken or `stop()` begins. That's no maintenance failure, and context-manager counts no attempt for it. Before, a resident's call made once `stop()` had begun was refused as a failure, so a restart that caught a maintenance tick before one of its calls logged a maintenance failure and raised a `context-maintenance-failed` alert; now the agent's maintenance logs that it stopped. An over-budget compile's `context-refusal` alert says whether the drain breaker kicked: `kicking`, `already draining`, or `not kicked` with the reason, where it said `kicking` whenever the agent was found. A merge keeps its place in the queue. A chunk's compression logs context-manager's "Failed to compress chunk (hierarchical)" line with the withdrawal, and the chunk is queued again at that manager's next rebuild, such as its next message or compile. The operator call never waits on a provider cooldown, or on the retry turn a cooldown queues, which the hold keeps from starting. `switchBranch` can therefore move the store after the call returns, and the API server's `branch.switch` announces the move it made once it has, while `branch.current`, the branch list, `/health` and `/status` read the store's branch whenever they report one. `maintenanceTick()` answers `ran: false` meanwhile, and `resume()` refuses with `OperatorActionError` (`agent-busy`), as the other operator calls do, including when the surgery took the store while `resume()` waited. Resume once the surgery has finished. diff --git a/src/framework.ts b/src/framework.ts index b96d41eaa..301f9189d 100644 --- a/src/framework.ts +++ b/src/framework.ts @@ -721,18 +721,20 @@ function logSeedOmissions( } /** - * A maintenance provider call withdrawn because a surgery holds the store. It - * sent nothing: it was refused at the provider gate, or woken there from its - * wait. context-manager counts no attempt for it. A merge keeps its queue - * entry, so the next tick after the hold does it again. A chunk's - * compression has already left the queue: it logs "Failed to compress chunk - * (hierarchical)" with this error and records the call, and the chunk, still - * uncompressed, is queued again at that manager's next rebuild (its next - * initialization, message, sent compile or released hold). + * A maintenance provider call withdrawn because a surgery holds the store, or + * because stop() has closed provider admission: either is the framework's own + * doing, and no failure. It sent nothing: it was refused at the provider + * gate, or woken there from its wait. context-manager counts no attempt for + * it. A merge keeps its queue entry, so the next tick after the hold, or + * after the next start, does it again. A chunk's compression has already + * left the queue: it logs "Failed to compress chunk (hierarchical)" with + * this error and records the call, and the chunk, still uncompressed, is + * queued again at that manager's next rebuild (its next initialization, + * message, sent compile or released hold). */ class MaintenanceWithdrawnError extends Error { - constructor(hold: { verb: string; agentName: string }) { - super(`maintenance call withdrawn: a live ${hold.verb} for ${hold.agentName} holds the store`); + constructor(reason: string) { + super(`maintenance call withdrawn: ${reason}`); this.name = 'MaintenanceWithdrawnError'; } } @@ -2854,9 +2856,7 @@ export class AgentFramework { // A surgery that took the store while this call was parked withdraws it. this.withdrawMaintenanceUnderSurgery(); } - if (this.providerAdmissionClosed) { - throw new Error(`Provider admission closed while stopping (${agentName})`); - } + this.withdrawMaintenanceWhenStopping(agentName); // A wait recorded while this call was parked binds it too. this.refuseHeldProviderModel(agentName, model); gate.auxiliaryInFlight++; @@ -2886,7 +2886,19 @@ export class AgentFramework { * cooldown queues, which the hold itself keeps from starting. */ private withdrawMaintenanceUnderSurgery(): void { - if (this.surgeryHold) throw new MaintenanceWithdrawnError(this.surgeryHold); + if (this.surgeryHold) { + throw new MaintenanceWithdrawnError(`a live ${this.surgeryHold.verb} for ${this.surgeryHold.agentName} holds the store`); + } + } + /** + * Nor once stop() has closed provider admission: a stop that catches a + * tick is the framework stopping on purpose, so the call is withdrawn, as + * under a surgery's hold, and no failure is reported for it. + */ + private withdrawMaintenanceWhenStopping(agentName: string): void { + if (this.providerAdmissionClosed) { + throw new MaintenanceWithdrawnError(`provider admission closed while stopping (${agentName})`); + } } private auxiliaryMembraneFor(agentName: string): Membrane { return this.membraneAdmitting((run, model) => this.withAuxiliaryAdmission(agentName, run, model)); @@ -2901,17 +2913,15 @@ export class AgentFramework { * recorded for it. (An ephemeral run records nothing, its name being * single-use, so its context manager keeps the plain membrane.) Its * compression is the background work a resident's is, whether a - * maintenance tick or a drain after a message starts it, so it meets the - * same refusals: a call made while a surgery holds the store is withdrawn, - * and one made once stop() has closed provider admission is refused. + * maintenance tick or a drain after a message starts it, so it is withdrawn + * as a resident's is: a call made while a surgery holds the store, or once + * stop() has closed provider admission, is withdrawn without being sent. */ private conversationAuxiliaryMembraneFor(agentName: string): Membrane { return this.membraneAdmitting(async (run, model) => { this.refuseHeldProviderModel(agentName, model); this.withdrawMaintenanceUnderSurgery(); - if (this.providerAdmissionClosed) { - throw new Error(`Provider admission closed while stopping (${agentName})`); - } + this.withdrawMaintenanceWhenStopping(agentName); try { return await run(); } catch (error) { if (model !== undefined && error instanceof Error) this.recordProviderWait(agentName, model, error); @@ -3494,9 +3504,10 @@ export class AgentFramework { const running = queued.filter((item): item is Extract => !item.failed); // Settled, not all: the pass ends only when every agent's work has. const outcomes = await Promise.allSettled(running.map(async ({ agent, cm, record }) => { - // A tick whose provider call a surgery withdrew ends this agent's - // work for the pass, as the hold would have before its next tick: it - // is no failure, and what the failure reports remember stays as it is. + // A tick whose provider call a surgery or stop() withdrew ends this + // agent's work for the pass, as the hold or the stop would have before + // its next tick: it is no failure, and what the failure reports + // remember stays as it is. let withdrawn = false; try { for ( @@ -5339,7 +5350,8 @@ export class AgentFramework { } catch (error) { // A manager that refuses its queries gives no verdict, as a preview // that can't be computed gives none: the agent's own state still says - // its budget and the patch's path. Anything else throws as before. + // its budget and the patch's path, and whatever stopped the preview is + // its reason. When the manager answers, anything else throws as before. if (agent.contextManagerAnswers()) throw error; return { available: false, reason: error instanceof Error ? error.message : String(error), ...agent.runtimeBudgetPath(patch) }; } @@ -7078,9 +7090,9 @@ export class AgentFramework { // before the later one began: from the earlier one's first stored // copy through the later one's first, each whole. Each end is every // copy of it besides, as the single form takes, so a copy stored - // outside that span (a write cut short and retried later) goes - // alone, and nothing stored after the later one began goes with it. - // Offline recovery's range is the same. + // outside that span (a write cut short and retried later) is taken + // on its own, and nothing else stored after the later one began is + // taken. Offline recovery's range is the same. const span = { from: bodyRangeIn(all, Math.min(fromIdx, toIdx)).from, to: bodyRangeIn(all, Math.max(fromIdx, toIdx)).to, @@ -15758,18 +15770,24 @@ export class AgentFramework { // this turn — surface it on the ops-alert pipeline (ops:alert trace + // webhook → fleet-watch) immediately, not only at the hard-down streak. // The per-(agent,kind) cooldown throttles repeats; failures.log already - // has the per-exhaustion record from logFailure above. + // has the per-exhaustion record from logFailure above. It says whether + // the drain breaker kicks, decided where the kick is. + const drainState = !agent ? 'unavailable (no agent handle)' + : !this.running ? 'not kicked: the framework is stopping' + : this.surgeryHold ? 'not kicked: a surgery holds the store' + : this.overBudgetDrainInFlight.has(agentName) ? 'already draining' + : 'kicking'; if (overBudget) { this.opsAlert( 'context-refusal', agentName, `compile refused (${errorType ?? 'unclassified'}): ${boundFailureText(wholeReason, FAILURE_SUMMARY_MAX_CHARS)} — ` + - `drain breaker ${agent ? 'kicking' : 'unavailable (no agent handle)'}; ` + + `drain breaker ${drainState}; ` + `if this persists, raise contextBudgetTokens or overBudgetGraceRatio.`, { skipLog: true }, ); } - if (agent && overBudget && this.running && !this.overBudgetDrainInFlight.has(agentName) && !this.surgeryHold) { + if (agent && overBudget && drainState === 'kicking') { this.overBudgetDrainInFlight.add(agentName); // Like a maintenance pass: none starts once stop() has begun, it starts // no tick while a surgery holds the store, and a branch move, or stop(), diff --git a/src/recovery/offline-branch.ts b/src/recovery/offline-branch.ts index 1bd3848b3..0b34243f7 100644 --- a/src/recovery/offline-branch.ts +++ b/src/recovery/offline-branch.ts @@ -445,8 +445,8 @@ function buildSuppressionPlan( // before the later one began: from the earlier one's first stored copy // through the later one's first. Each end is every copy of it besides, // as a single suppression is, so a copy stored outside that span (a - // write cut short and retried later) goes alone, and nothing stored - // after the later one began goes with it. + // write cut short and retried later) is taken on its own, and nothing + // else stored after the later one began is taken. intervals.push({ start: earlier.first, end: later.first, diff --git a/test/host-quiesce.test.ts b/test/host-quiesce.test.ts index da7eba3ae..c260dd301 100644 --- a/test/host-quiesce.test.ts +++ b/test/host-quiesce.test.ts @@ -341,6 +341,15 @@ test('resume proceeds without a verdict for an agent whose context manager refus assert.equal(preview.effective, undefined); // A patch whose values the agent refuses still throws. assert.throws(() => framework.previewAgentRuntimeSettings('agent', { contextBudgetTokens: 5 }), /contextBudgetTokens must be/); + assert.throws(() => framework.previewAgentRuntimeSettings('agent', { tailTokens: -1 }), /tailTokens must be/); + // Whether the strategy takes a live tail or pace is the manager's to say, + // so while it refuses, a patch that sets one gets no verdict either. + for (const patch of [{ tailTokens: 2_000 }, { transitionPaceTokens: 1_000 }]) { + const unanswered = framework.previewAgentRuntimeSettings('agent', patch); + assert.equal(unanswered.available, false, JSON.stringify(patch)); + assert.match(String(unanswered.reason), /requires successful strategy initialization/); + assert.equal(unanswered.liveBudgetTokens, 100_000); + } // An update of the budget is refused, by the manager, and the preflight // doesn't say it's applying one. diff --git a/test/maintenance-timer.test.ts b/test/maintenance-timer.test.ts index 5bc071cd0..bf8aee487 100644 --- a/test/maintenance-timer.test.ts +++ b/test/maintenance-timer.test.ts @@ -360,7 +360,6 @@ class CallAfterGateStrategy implements ContextStrategy { ticks = 0; entered = false; outcome?: string; - outcomeMessage?: string; private open?: () => void; private readonly gate = new Promise((resolve) => { this.open = resolve; }); @@ -385,7 +384,6 @@ class CallAfterGateStrategy implements ContextStrategy { this.outcome = 'sent'; } catch (error) { this.outcome = (error as Error).name; - this.outcomeMessage = (error as Error).message; throw error; } } @@ -1352,43 +1350,56 @@ describe('queued context maintenance timer', () => { } }); - it("a conversation agent's call made once stop() has begun is refused, as a resident's is", { timeout: 10_000 }, async () => { - const dir = mkdtempSync(join(tmpdir(), 'af-maintenance-conversation-stopping-')); - const provider = new CapturingMembrane(); - const conversation = new CallAfterGateStrategy(); - const framework = await AgentFramework.create({ - storePath: join(dir, 'store'), - membrane: provider.asMembrane(), - agents: [{ name: 'agent', model: 'test-model', systemPrompt: 'test', strategy: new ReadyStrategy() }], - modules: [], - syncIntervalMs: 0, - maintenanceIntervalMs: 10, - conversations: { templateAgent: 'agent', strategyFactory: () => conversation }, + for (const kind of ['resident', 'conversation agent'] as const) { + it(`a ${kind}'s maintenance call made once stop() has begun is withdrawn, and no failure is reported`, { timeout: 10_000 }, async () => { + const dir = mkdtempSync(join(tmpdir(), 'af-maintenance-stopping-')); + const provider = new CapturingMembrane(); + const tick = new CallAfterGateStrategy(); + const conversational = kind === 'conversation agent'; + const framework = await AgentFramework.create({ + storePath: join(dir, 'store'), + membrane: provider.asMembrane(), + agents: [{ name: 'agent', model: 'test-model', systemPrompt: 'test', strategy: conversational ? new ReadyStrategy() : tick }], + modules: [], + syncIntervalMs: 0, + maintenanceIntervalMs: 10, + ...(conversational ? { conversations: { templateAgent: 'agent', strategyFactory: () => tick } } : {}), + }); + const internals = framework as unknown as { + createConversationAgent(name: string, channelId: string): Promise; + }; + const name = conversational ? 'agent-conv-1' : 'agent'; + const alerts: string[] = []; + (framework as any).opsAlert = (...args: unknown[]) => { alerts.push(args.map(String).join(' ')); }; + const quiet = console.error; + const logged: string[] = []; + console.error = (...args: unknown[]) => { logged.push(args.map(String).join(' ')); }; + let stopping: Promise | undefined; + try { + if (conversational) await internals.createConversationAgent(name, 'chan-1'); + framework.start(); + await waitFor(() => tick.entered); + stopping = framework.stop(); + tick.release(); + await waitFor(() => tick.outcome !== undefined); + await stopping; + assert.equal(tick.outcome, 'MaintenanceWithdrawnError'); + assert.equal(provider.calls.length, 0, 'the withdrawn call sent nothing'); + assert.ok(logged.some((line) => line.includes( + `[context-maintenance] agent=${name} stopped: maintenance call withdrawn: provider admission closed while stopping (${name})`, + )), logged.join('\n')); + assert.deepEqual(alerts, [], 'no alert'); + for (const run of framework.getContextMaintenanceSnapshot().history) { + for (const record of run.agents) assert.equal(record.error, undefined, `no failure recorded for ${record.agentName}`); + } + } finally { + console.error = quiet; + tick.release(); + await (stopping ?? framework.stop()); + rmSync(dir, { recursive: true, force: true }); + } }); - const internals = framework as unknown as { - createConversationAgent(name: string, channelId: string): Promise; - }; - (framework as any).opsAlert = () => {}; - const quiet = console.error; - console.error = () => {}; - let stopping: Promise | undefined; - try { - await internals.createConversationAgent('agent-conv-1', 'chan-1'); - framework.start(); - await waitFor(() => conversation.entered); - stopping = framework.stop(); - conversation.release(); - await waitFor(() => conversation.outcome !== undefined); - assert.match(conversation.outcomeMessage ?? '', /Provider admission closed while stopping \(agent-conv-1\)/); - assert.equal(provider.calls.length, 0, 'the refused call sent nothing'); - await stopping; - } finally { - console.error = quiet; - conversation.release(); - await (stopping ?? framework.stop()); - rmSync(dir, { recursive: true, force: true }); - } - }); + } it('no over-budget drain kick starts once stop() has begun', async () => { const dir = mkdtempSync(join(tmpdir(), 'af-maintenance-drain-after-stop-')); @@ -1404,7 +1415,8 @@ describe('queued context maintenance timer', () => { noteInferenceExhausted(agentName: string, reason: string, retryable?: boolean, errorType?: string): void; overBudgetDrains: Set>; }; - (framework as any).opsAlert = () => {}; + const alerts: string[] = []; + (framework as any).opsAlert = (...args: unknown[]) => { alerts.push(args.map(String).join(' ')); }; const quiet = console.error; const logged: string[] = []; console.error = (...args: unknown[]) => { logged.push(args.map(String).join(' ')); }; @@ -1413,6 +1425,8 @@ describe('queued context maintenance timer', () => { const stopping = framework.stop(); internals.noteInferenceExhausted('agent', 'compile refused', false, 'over_budget'); assert.equal(internals.overBudgetDrains.size, 0, 'no kick was started'); + assert.ok(alerts.some((alert) => alert.startsWith('context-refusal agent ') + && alert.includes('drain breaker not kicked: the framework is stopping;')), alerts.join('\n')); await stopping; assert.ok(!logged.some((line) => line.includes('drain kicked')), logged.join('\n')); } finally { diff --git a/test/offline-recovery-branch.test.ts b/test/offline-recovery-branch.test.ts index bd2a61a6b..24e7bf453 100644 --- a/test/offline-recovery-branch.test.ts +++ b/test/offline-recovery-branch.test.ts @@ -241,6 +241,47 @@ test('a suppressed range takes every copy of both its messages, and leaves what } }); +test('a suppressed range whose two messages are stored in shards takes each whole', async () => { + const dir = mkdtempSync(join(tmpdir(), 'offline-recovery-range-shards-')); + const storePath = join(dir, 'agent.chronicle'); + try { + const store = JsStore.openOrCreate({ path: storePath }); + const cm = await ContextManager.open({ store, namespace: 'agents/cairn' }); + const add = (text: string, messageId: string) => cm.addMessage('user', [{ type: 'text', text }], { + serverId: 'discord', channelId: 'discord:g1:c1', messageId, + }); + add('keep', 'm-keep'); + // Each shard of a long message carries its Discord message id. + appendShards(cm, 'body-a', ['m-a', 'm-a']); + add('between', 'm-between'); + appendShards(cm, 'body-b', ['m-b', 'm-b', 'm-b']); + add('after', 'm-after'); + add('current anchor', 'm-current'); + cm.close(); + store.close(); + + const result = await createOfflineRecoveryBranch({ + storePath, + agentName: 'cairn', + messageId: 'm-current', + suppressRanges: [{ fromMessageId: 'm-b', toMessageId: 'm-a' }], + branchName: 'recovery/cairn/range-shards', + }); + + assert.equal(result.messagesSuppressed, 6, 'both bodies, every shard, and what lies between'); + const recoveredStore = JsStore.openOrCreate({ path: storePath }); + const recoveredCm = await ContextManager.open({ store: recoveredStore, namespace: 'agents/cairn' }); + assert.deepEqual( + recoveredCm.getAllMessages().map((message) => message.metadata?.messageId), + ['m-keep', 'm-after', 'm-current'], + ); + recoveredCm.close(); + recoveredStore.close(); + } finally { + rmSync(dir, { recursive: true, force: true }); + } +}); + test('offline recovery accepts an exact internal context ID anchor', async () => { const dir = mkdtempSync(join(tmpdir(), 'offline-recovery-context-id-')); const storePath = join(dir, 'agent.chronicle'); From fe9da2fc8b9c484f27b342aabc11af85c8b5b971 Mon Sep 17 00:00:00 2001 From: Gladys-2413 Date: Thu, 15 Oct 2026 18:23:00 -0400 Subject: [PATCH 04/11] fix(maintenance): drains in a framework never started, and a stop over a held model Greptile's review of #377 found two things, each fixed here. The over-budget drain kick, and its loop, checked `running` to leave stop() alone. A framework run with runUntilIdle(), never started, has `running` false and no maintenance timer, so its agents lost the drain, the one recovery left to them, and the context-refusal alert said the framework was stopping. What ends drains is stop(), so the kick, its loop and the alert now read stop()'s own flag, provider admission closed, which start() opens and stop() closes. The two test harnesses that set `running` to keep the drain no longer need to. A maintenance call whose model a provider wait holds was refused for the wait before the surgery's or the stop's withdrawal was checked, so a tick that met a held model under a surgery's hold, or once stop() had begun, was reported as a maintenance failure and alerted. Both membranes now withdraw such a call first, and the resident's checks a stop before it parks at the gate as well as after. Tests, in maintenance-timer: a resident's and a conversation agent's call under a surgery's hold, and once stop() has begun, each with and without a provider wait on its model, is withdrawn with no failure reported; stop() withdraws a call parked at the provider gate; an over-budget drain kicks in a framework that was never started and ticks its eight ticks. --- src/framework.ts | 17 ++- test/inference-failure-bounds.test.ts | 2 - test/maintenance-timer.test.ts | 168 +++++++++++++++++++------- test/overbudget-drain-breaker.test.ts | 2 - 4 files changed, 136 insertions(+), 53 deletions(-) diff --git a/src/framework.ts b/src/framework.ts index 301f9189d..c4036a7d5 100644 --- a/src/framework.ts +++ b/src/framework.ts @@ -2840,8 +2840,11 @@ export class AgentFramework { } private async withAuxiliaryAdmission(agentName: string, run: () => Promise, model?: string): Promise { const gate = this.providerGate(agentName); - this.refuseHeldProviderModel(agentName, model); + // A surgery's hold or a stop withdraws the call before a provider wait + // can refuse it: either is the framework's own doing, and no failure. this.withdrawMaintenanceUnderSurgery(); + this.withdrawMaintenanceWhenStopping(agentName); + this.refuseHeldProviderModel(agentName, model); // A parked call is woken when the gate opens, when stop() closes // admission, or when a surgery takes the store; one woken while the gate // is still closed parks again. @@ -2919,9 +2922,9 @@ export class AgentFramework { */ private conversationAuxiliaryMembraneFor(agentName: string): Membrane { return this.membraneAdmitting(async (run, model) => { - this.refuseHeldProviderModel(agentName, model); this.withdrawMaintenanceUnderSurgery(); this.withdrawMaintenanceWhenStopping(agentName); + this.refuseHeldProviderModel(agentName, model); try { return await run(); } catch (error) { if (model !== undefined && error instanceof Error) this.recordProviderWait(agentName, model, error); @@ -15771,9 +15774,11 @@ export class AgentFramework { // webhook → fleet-watch) immediately, not only at the hard-down streak. // The per-(agent,kind) cooldown throttles repeats; failures.log already // has the per-exhaustion record from logFailure above. It says whether - // the drain breaker kicks, decided where the kick is. + // the drain breaker kicks, decided where the kick is. What ends drains is + // stop(), not a framework that was never started: one run with + // runUntilIdle() has no maintenance timer, so a drain is its recovery. const drainState = !agent ? 'unavailable (no agent handle)' - : !this.running ? 'not kicked: the framework is stopping' + : this.providerAdmissionClosed ? 'not kicked: the framework is stopping' : this.surgeryHold ? 'not kicked: a surgery holds the store' : this.overBudgetDrainInFlight.has(agentName) ? 'already draining' : 'kicking'; @@ -15796,11 +15801,11 @@ export class AgentFramework { let ticks = 0; try { const cm = agent.getContextManager(); - while (ticks < 8 && this.running && !this.surgeryHold) { + while (ticks < 8 && !this.providerAdmissionClosed && !this.surgeryHold) { await cm.tick(); ticks++; } - const stopped = ticks >= 8 ? '' : !this.running ? ', stopped as the framework stops' : ', stopped while a surgery holds the store'; + const stopped = ticks >= 8 ? '' : this.providerAdmissionClosed ? ', stopped as the framework stops' : ', stopped while a surgery holds the store'; console.error(`[inference-failed] drain kicked for ${agentName} (OverBudget breaker, ${ticks} ticks${stopped})`); } catch (err) { if (err instanceof MaintenanceWithdrawnError) { diff --git a/test/inference-failure-bounds.test.ts b/test/inference-failure-bounds.test.ts index 352f6e498..6fe63cdbe 100644 --- a/test/inference-failure-bounds.test.ts +++ b/test/inference-failure-bounds.test.ts @@ -55,8 +55,6 @@ function makeHarness(opts?: { tick?: () => Promise; addMessage?: (p: strin fw.pendingRequests = []; fw.overBudgetDrainInFlight = new Set(); fw.overBudgetDrains = new Set(); - // A drain kick ticks only while the framework runs. - fw.running = true; fw.opsAlertLastSent = new Map(); fw.opsAlertCooldownMs = 15 * 60_000; const traces: any[] = []; diff --git a/test/maintenance-timer.test.ts b/test/maintenance-timer.test.ts index bf8aee487..650ffc677 100644 --- a/test/maintenance-timer.test.ts +++ b/test/maintenance-timer.test.ts @@ -25,6 +25,7 @@ import type { import { AgentFramework } from '../src/index.js'; import { OperatorActionError } from '../src/operator-log.js'; import type { NormalizedRequest, NormalizedResponse } from '@animalabs/membrane'; +import { MembraneError } from '@animalabs/membrane'; class QueuedStrategy implements ContextStrategy { readonly name = 'queued-test'; @@ -354,6 +355,14 @@ class ParkedCallTickStrategy implements ContextStrategy { } } +/** A provider's stated wait of `ms`, as membrane reports a 429 with retry-after. */ +function heldFor(ms: number): MembraneError { + return new MembraneError({ + type: 'rate_limit', retryable: true, httpStatus: 429, retryAfterMs: ms, + message: 'rate limited for the test', rawError: { status: 429 }, + }); +} + /** A strategy whose first tick waits on a gate, then calls its manager's membrane, as a compression would. */ class CallAfterGateStrategy implements ContextStrategy { readonly name = 'call-after-gate-test'; @@ -379,6 +388,7 @@ class CallAfterGateStrategy implements ContextStrategy { try { await ctx.membrane?.complete({ model: 'test-model', + config: { model: 'test-model' }, messages: [{ role: 'user', content: [{ type: 'text', text: 'compress' }] }], } as unknown as NormalizedRequest); this.outcome = 'sent'; @@ -1249,48 +1259,56 @@ describe('queued context maintenance timer', () => { } }); - it("a conversation agent's tick that reaches its provider under a surgery's hold is withdrawn, as a resident's is", { timeout: 10_000 }, async () => { - const dir = mkdtempSync(join(tmpdir(), 'af-maintenance-conversation-withdrawn-')); - const provider = new CapturingMembrane(); - const conversation = new CallAfterGateStrategy(); - const framework = await AgentFramework.create({ - storePath: join(dir, 'store'), - membrane: provider.asMembrane(), - agents: [{ name: 'agent', model: 'test-model', systemPrompt: 'test', strategy: new ReadyStrategy() }], - modules: [], - syncIntervalMs: 0, - maintenanceIntervalMs: 10, - conversations: { templateAgent: 'agent', strategyFactory: () => conversation }, - }); - const internals = framework as unknown as { - createConversationAgent(name: string, channelId: string): Promise; - reserveStoreForSurgery(verb: string, agentName: string): () => void; - }; - const alerts: string[] = []; - (framework as any).opsAlert = (kind: string, agentName: string) => alerts.push(`${kind}:${agentName}`); - const quiet = console.error; - console.error = () => {}; - try { - await internals.createConversationAgent('agent-conv-1', 'chan-1'); - framework.start(); - await waitFor(() => conversation.entered); - const release = internals.reserveStoreForSurgery('roll back', 'agent'); - try { - conversation.release(); - await waitFor(() => conversation.outcome !== undefined); - assert.equal(conversation.outcome, 'MaintenanceWithdrawnError'); - assert.equal(provider.calls.length, 0, 'the withdrawn call sent nothing'); - } finally { - release(); - } - assert.deepEqual(alerts.filter((alert) => alert.startsWith('context-maintenance-failed')), []); - } finally { - console.error = quiet; - conversation.release(); - await framework.stop(); - rmSync(dir, { recursive: true, force: true }); + for (const kind of ['resident', 'conversation agent'] as const) { + for (const held of [false, true]) { + it(`a ${kind}'s tick that reaches its provider under a surgery's hold is withdrawn${held ? ', though a provider wait holds its model' : ''}`, { timeout: 10_000 }, async () => { + const dir = mkdtempSync(join(tmpdir(), 'af-maintenance-surgery-withdrawn-')); + const provider = new CapturingMembrane(); + const tick = new CallAfterGateStrategy(); + const conversational = kind === 'conversation agent'; + const framework = await AgentFramework.create({ + storePath: join(dir, 'store'), + membrane: provider.asMembrane(), + agents: [{ name: 'agent', model: 'test-model', systemPrompt: 'test', strategy: conversational ? new ReadyStrategy() : tick }], + modules: [], + syncIntervalMs: 0, + maintenanceIntervalMs: 10, + ...(conversational ? { conversations: { templateAgent: 'agent', strategyFactory: () => tick } } : {}), + }); + const internals = framework as unknown as { + createConversationAgent(name: string, channelId: string): Promise; + reserveStoreForSurgery(verb: string, agentName: string): () => void; + recordProviderWait(agentName: string, model: string, error: Error): unknown; + }; + const name = conversational ? 'agent-conv-1' : 'agent'; + const alerts: string[] = []; + (framework as any).opsAlert = (kind: string, agentName: string) => alerts.push(`${kind}:${agentName}`); + const quiet = console.error; + console.error = () => {}; + try { + if (conversational) await internals.createConversationAgent(name, 'chan-1'); + if (held) internals.recordProviderWait(name, 'test-model', heldFor(60_000)); + framework.start(); + await waitFor(() => tick.entered); + const release = internals.reserveStoreForSurgery('roll back', 'agent'); + try { + tick.release(); + await waitFor(() => tick.outcome !== undefined); + assert.equal(tick.outcome, 'MaintenanceWithdrawnError'); + assert.equal(provider.calls.length, 0, 'the withdrawn call sent nothing'); + } finally { + release(); + } + assert.deepEqual(alerts.filter((alert) => alert.startsWith('context-maintenance-failed')), []); + } finally { + console.error = quiet; + tick.release(); + await framework.stop(); + rmSync(dir, { recursive: true, force: true }); + } + }); } - }); + } it('a pass that only meets a failure already reported stays out of the history', { timeout: 10_000 }, async () => { const dir = mkdtempSync(join(tmpdir(), 'af-maintenance-repeats-')); @@ -1350,8 +1368,8 @@ describe('queued context maintenance timer', () => { } }); - for (const kind of ['resident', 'conversation agent'] as const) { - it(`a ${kind}'s maintenance call made once stop() has begun is withdrawn, and no failure is reported`, { timeout: 10_000 }, async () => { + for (const kind of ['resident', 'conversation agent'] as const) for (const held of [false, true]) { + it(`a ${kind}'s maintenance call made once stop() has begun is withdrawn${held ? ', though a provider wait holds its model,' : ''} and no failure is reported`, { timeout: 10_000 }, async () => { const dir = mkdtempSync(join(tmpdir(), 'af-maintenance-stopping-')); const provider = new CapturingMembrane(); const tick = new CallAfterGateStrategy(); @@ -1367,6 +1385,7 @@ describe('queued context maintenance timer', () => { }); const internals = framework as unknown as { createConversationAgent(name: string, channelId: string): Promise; + recordProviderWait(agentName: string, model: string, error: Error): unknown; }; const name = conversational ? 'agent-conv-1' : 'agent'; const alerts: string[] = []; @@ -1377,6 +1396,7 @@ describe('queued context maintenance timer', () => { let stopping: Promise | undefined; try { if (conversational) await internals.createConversationAgent(name, 'chan-1'); + if (held) internals.recordProviderWait(name, 'test-model', heldFor(60_000)); framework.start(); await waitFor(() => tick.entered); stopping = framework.stop(); @@ -1401,6 +1421,68 @@ describe('queued context maintenance timer', () => { }); } + it('stop() withdraws a maintenance call parked at the provider gate, and that is no failure', { timeout: 10_000 }, async () => { + const dir = mkdtempSync(join(tmpdir(), 'af-maintenance-parked-stop-')); + const strategy = new ParkedCallTickStrategy(); + const framework = await AgentFramework.create({ + storePath: join(dir, 'store'), + membrane, + agents: [{ name: 'agent', model: 'test-model', systemPrompt: 'test', strategy }], + modules: [], + syncIntervalMs: 0, + maintenanceIntervalMs: 10, + }); + strategy.framework = framework; + const alerts: string[] = []; + (framework as any).opsAlert = (...args: unknown[]) => { alerts.push(args.map(String).join(' ')); }; + const internals = framework as unknown as { providerGate(name: string): { auxiliaryWaiters: unknown[] } }; + const quiet = console.error; + console.error = () => {}; + try { + framework.start(); + await waitFor(() => internals.providerGate('agent').auxiliaryWaiters.length === 1); + await framework.stop(); + assert.equal(strategy.outcome, 'MaintenanceWithdrawnError'); + assert.equal(strategy.calls, 0, 'the withdrawn call sent nothing'); + assert.deepEqual(alerts, []); + } finally { + console.error = quiet; + rmSync(dir, { recursive: true, force: true }); + } + }); + + it('an over-budget drain kick starts in a framework that was never started, as one run with runUntilIdle() is', async () => { + const dir = mkdtempSync(join(tmpdir(), 'af-maintenance-drain-unstarted-')); + const framework = await AgentFramework.create({ + storePath: join(dir, 'store'), + membrane, + agents: [{ name: 'agent', model: 'test-model', systemPrompt: 'test', strategy: new ReadyStrategy() }], + modules: [], + syncIntervalMs: 0, + maintenanceIntervalMs: 0, + }); + const internals = framework as unknown as { + noteInferenceExhausted(agentName: string, reason: string, retryable?: boolean, errorType?: string): void; + overBudgetDrains: Set>; + }; + const alerts: string[] = []; + (framework as any).opsAlert = (...args: unknown[]) => { alerts.push(args.map(String).join(' ')); }; + const quiet = console.error; + const logged: string[] = []; + console.error = (...args: unknown[]) => { logged.push(args.map(String).join(' ')); }; + try { + internals.noteInferenceExhausted('agent', 'compile refused', false, 'over_budget'); + assert.equal(internals.overBudgetDrains.size, 1, 'a kick was started'); + assert.ok(alerts.some((alert) => alert.includes('drain breaker kicking;')), alerts.join('\n')); + await Promise.all([...internals.overBudgetDrains]); + assert.ok(logged.some((line) => line.includes('drain kicked for agent (OverBudget breaker, 8 ticks)')), logged.join('\n')); + } finally { + console.error = quiet; + await framework.stop(); + rmSync(dir, { recursive: true, force: true }); + } + }); + it('no over-budget drain kick starts once stop() has begun', async () => { const dir = mkdtempSync(join(tmpdir(), 'af-maintenance-drain-after-stop-')); const framework = await AgentFramework.create({ diff --git a/test/overbudget-drain-breaker.test.ts b/test/overbudget-drain-breaker.test.ts index f18a589a6..3defc651e 100644 --- a/test/overbudget-drain-breaker.test.ts +++ b/test/overbudget-drain-breaker.test.ts @@ -33,8 +33,6 @@ function makeHarness(tick: () => Promise) { fw.lastInferenceAt = new Map(); fw.overBudgetDrainInFlight = new Set(); fw.overBudgetDrains = new Set(); - // A drain kick ticks only while the framework runs. - fw.running = true; fw.opsAlertLastSent = new Map(); fw.pendingRequests = []; fw.traceListeners = []; From 949de457ab60086fb2237fd1cf6d2a70b97c29a7 Mon Sep 17 00:00:00 2001 From: Gladys-2413 Date: Thu, 15 Oct 2026 18:31:48 -0400 Subject: [PATCH 05/11] fix(inference): decide the drain breaker's state only for a refusal over budget The previous commit read stop()'s own flag for the drain's state, and the state was decided for every exhausted inference, over budget or not, so a harness built without the drain's sets met an undefined one and threw: 14 tests across inference-exhaustion-rewind, inference-failure-observability and ops-alert. Before, an unset `running` had stopped the decision first. Only a refusal over budget has a drain to decide, so the state is decided only then. --- src/framework.ts | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/framework.ts b/src/framework.ts index c4036a7d5..470fda673 100644 --- a/src/framework.ts +++ b/src/framework.ts @@ -15777,7 +15777,8 @@ export class AgentFramework { // the drain breaker kicks, decided where the kick is. What ends drains is // stop(), not a framework that was never started: one run with // runUntilIdle() has no maintenance timer, so a drain is its recovery. - const drainState = !agent ? 'unavailable (no agent handle)' + const drainState = !overBudget ? undefined + : !agent ? 'unavailable (no agent handle)' : this.providerAdmissionClosed ? 'not kicked: the framework is stopping' : this.surgeryHold ? 'not kicked: a surgery holds the store' : this.overBudgetDrainInFlight.has(agentName) ? 'already draining' From c32b69efc7ff08cf5f5bfaaea5a61045a39712c9 Mon Sep 17 00:00:00 2001 From: Gladys-2413 Date: Thu, 15 Oct 2026 23:13:33 -0400 Subject: [PATCH 06/11] fix(maintenance): the points of Anarchid's review of #377 R1-1. A withdrawal that lands inside context-manager 0.14's compression fallback ladder, after a chunk's first call was refused or cut short, is caught by the rung that made it and recorded as a provider error: 0.14 rethrows only its own CompressionBranchDiscard there. Once every rung has failed, the chunk's request family is quarantined, with its alert, and the tick returns normally. The withdrawal's doc and the changelog said context-manager counts no attempt for a withdrawn call; they now say that holds for a merge's call and a chunk's first, and what happens inside the ladder, until context-manager rethrows such a withdrawal. R1-2. An operator's maintain during a lasting failure answered ran: true, but the history rule dropped the pass it ran, so the newest run was an older report. A pass maintenanceTick() asked for is now kept. R1-3. The refusing manager's preview returned the raw message, so a stale branch generation never came back as the documented branch_generation_changed. One helper now maps a preview's known refusals for both catches. R1-4. The over-budget drain's body ran before it joined the drains stop() waits for, so a stop() made from inside its first tick, before that tick's first await, didn't wait for the tick and could close the store under it. The drain now joins first and runs after. Its decision is also read again after the refusal's alert, so a stop() or a surgery begun by an ops:alert listener starts no kick. R1-5. A test comment said a drain kick ticks only while the framework runs, which stopped being true with the drain's own stop flag; the comment and the start() under it are gone. Agent's runtimeBudgetPath, contextManagerAnswers and checkRuntimeSettingsPatch are marked @internal: they serve the framework's own preview, not hosts. Tests: in maintenance-timer, an operator's maintain keeps the pass it ran though it only met a failure already reported; a stop() begun from inside the refusal's alert starts no drain kick; a stop() made from a drain's first tick waits for that tick. In host-quiesce, a preview stopped by a stale branch generation says branch_generation_changed. --- changelog.d/cm-0.14.changed.md | 4 +- src/agent.ts | 7 +- src/framework.ts | 63 +++++++++++------ test/host-quiesce.test.ts | 25 +++++++ test/maintenance-timer.test.ts | 122 ++++++++++++++++++++++++++++++++- 5 files changed, 194 insertions(+), 27 deletions(-) diff --git a/changelog.d/cm-0.14.changed.md b/changelog.d/cm-0.14.changed.md index 9abed625c..6dea6b12e 100644 --- a/changelog.d/cm-0.14.changed.md +++ b/changelog.d/cm-0.14.changed.md @@ -1,2 +1,2 @@ -- A context-maintenance pass no longer fails as a whole when one agent's readiness can't be read. context-manager 0.14, which agent-framework has required since 0.21.1, refuses its readiness queries while a strategy's initialization is pending (inside `switchBranch`, `fork`, `setStrategy` or `ContextManager.open`) or has failed, and a strategy can refuse on its own, as the autobiographical one does for a branch it hasn't reloaded. That agent's run now records the error, the agent gets its own `context-maintenance-failed` alert (`scope: 'agent'`), and the other agents run. Before, such a throw ended the pass for every agent, with one alert for the whole pass (`scope: 'pass'`). Anything else that throws while one agent's maintenance is prepared is that agent's failure in the same way. A failure that lasts, met by every pass, is logged, alerted and recorded in the maintenance history when it's first met, then again when its text changes, every 15 minutes while it lasts (the alert webhook's cooldown for it, counted from the webhook's last post of it), or after that agent's maintenance has gone without a failure. A pass that met only failures already reported, and took no tick, isn't added to the history, so a lasting failure no longer pushes out the runs that did work. `getContextMaintenanceSnapshot()` reports such an agent as `ready: false` with an `error`, and a strategy that can't give a progress snapshot reads as having none. -- While an operator surgery (`rollbackToMessage` or `suppressMessages`) or `switchBranch` holds the store, no maintenance pass starts and no over-budget drain kick is started, a running pass or drain kick starts no new tick, and the operator call waits for them to finish before it switches or forks the branch, so no maintenance work overlaps the move. `stop()` waits for them the same way, a drain kick included, before it closes the store, so the tick in flight ends before the store closes, and once `stop()` has begun, no drain kick starts and none starts a new tick. Maintenance makes no provider call meanwhile, a conversation agent's included: one made under the hold, or once `stop()` has begun, is withdrawn without being sent, and so is a resident's call waiting at its provider gate when the hold is taken or `stop()` begins. That's no maintenance failure, and context-manager counts no attempt for it. Before, a resident's call made once `stop()` had begun was refused as a failure, so a restart that caught a maintenance tick before one of its calls logged a maintenance failure and raised a `context-maintenance-failed` alert; now the agent's maintenance logs that it stopped. An over-budget compile's `context-refusal` alert says whether the drain breaker kicked: `kicking`, `already draining`, or `not kicked` with the reason, where it said `kicking` whenever the agent was found. A merge keeps its place in the queue. A chunk's compression logs context-manager's "Failed to compress chunk (hierarchical)" line with the withdrawal, and the chunk is queued again at that manager's next rebuild, such as its next message or compile. The operator call never waits on a provider cooldown, or on the retry turn a cooldown queues, which the hold keeps from starting. `switchBranch` can therefore move the store after the call returns, and the API server's `branch.switch` announces the move it made once it has, while `branch.current`, the branch list, `/health` and `/status` read the store's branch whenever they report one. `maintenanceTick()` answers `ran: false` meanwhile, and `resume()` refuses with `OperatorActionError` (`agent-busy`), as the other operator calls do, including when the surgery took the store while `resume()` waited. Resume once the surgery has finished. +- A context-maintenance pass no longer fails as a whole when one agent's readiness can't be read. context-manager 0.14, which agent-framework has required since 0.21.1, refuses its readiness queries while a strategy's initialization is pending (inside `switchBranch`, `fork`, `setStrategy` or `ContextManager.open`) or has failed, and a strategy can refuse on its own, as the autobiographical one does for a branch it hasn't reloaded. That agent's run now records the error, the agent gets its own `context-maintenance-failed` alert (`scope: 'agent'`), and the other agents run. Before, such a throw ended the pass for every agent, with one alert for the whole pass (`scope: 'pass'`). Anything else that throws while one agent's maintenance is prepared is that agent's failure in the same way. A failure that lasts, met by every pass, is logged, alerted and recorded in the maintenance history when it's first met, then again when its text changes, every 15 minutes while it lasts (the alert webhook's cooldown for it, counted from the webhook's last post of it), or after that agent's maintenance has gone without a failure. A pass that met only failures already reported, and took no tick, isn't added to the history unless an operator's `maintain` (`maintenanceTick()`) asked for it, so a lasting failure no longer pushes out the runs that did work. `getContextMaintenanceSnapshot()` reports such an agent as `ready: false` with an `error`, and a strategy that can't give a progress snapshot reads as having none. +- While an operator surgery (`rollbackToMessage` or `suppressMessages`) or `switchBranch` holds the store, no maintenance pass starts and no over-budget drain kick is started, a running pass or drain kick starts no new tick, and the operator call waits for them to finish before it switches or forks the branch, so no maintenance work overlaps the move. `stop()` waits for them the same way, a drain kick included, before it closes the store, so the tick in flight ends before the store closes, and once `stop()` has begun, no drain kick starts and none starts a new tick. Maintenance makes no provider call meanwhile, a conversation agent's included: one made under the hold, or once `stop()` has begun, is withdrawn without being sent, and so is a resident's call waiting at its provider gate when the hold is taken or `stop()` begins. That's no maintenance failure, and context-manager counts no attempt for it, where the withdrawal is a merge's call or a chunk's first. One inside context-manager 0.14's compression fallback ladder, after a chunk's first call was refused or cut short, is recorded there as a provider error, and once every rung has failed, the chunk is quarantined, with its alert, until context-manager rethrows such a withdrawal. Before, a resident's call made once `stop()` had begun was refused as a failure, so a restart that caught a maintenance tick before one of its calls logged a maintenance failure and raised a `context-maintenance-failed` alert; now the agent's maintenance logs that it stopped. An over-budget compile's `context-refusal` alert says whether the drain breaker kicked: `kicking`, `already draining`, or `not kicked` with the reason, where it said `kicking` whenever the agent was found. A merge keeps its place in the queue. A chunk's compression logs context-manager's "Failed to compress chunk (hierarchical)" line with the withdrawal, and the chunk is queued again at that manager's next rebuild, such as its next message or compile. The operator call never waits on a provider cooldown, or on the retry turn a cooldown queues, which the hold keeps from starting. `switchBranch` can therefore move the store after the call returns, and the API server's `branch.switch` announces the move it made once it has, while `branch.current`, the branch list, `/health` and `/status` read the store's branch whenever they report one. `maintenanceTick()` answers `ran: false` meanwhile, and `resume()` refuses with `OperatorActionError` (`agent-busy`), as the other operator calls do, including when the surgery took the store while `resume()` waited. Resume once the surgery has finished. diff --git a/src/agent.ts b/src/agent.ts index 01d26752a..7e12d3636 100644 --- a/src/agent.ts +++ b/src/agent.ts @@ -501,6 +501,7 @@ export class Agent { * whether a descent is in flight, from the agent's own state alone: what a * runtime-settings preview can still say while the context manager refuses * its queries. + * @internal For the framework's runtime-settings preview; not a host API. */ runtimeBudgetPath(patch?: AgentRuntimeSettingsPatch): { liveBudgetTokens: number; @@ -522,6 +523,7 @@ export class Agent { * failed, until a switchBranch or setStrategy succeeds, and a strategy can * refuse its readiness on its own, as the autobiographical one does for a * branch it hasn't reloaded. + * @internal For the framework's runtime-settings preview; not a host API. */ contextManagerAnswers(): boolean { try { @@ -799,7 +801,10 @@ export class Agent { } } - /** The checks of a runtime-settings patch that need nothing from the context manager. */ + /** + * The checks of a runtime-settings patch that need nothing from the context manager. + * @internal For the framework's runtime-settings preview; not a host API. + */ checkRuntimeSettingsPatch(patch: AgentRuntimeSettingsPatch): void { if (Object.keys(patch).length === 0) throw new Error('At least one setting is required'); if (patch.contextBudgetTokens !== undefined) { diff --git a/src/framework.ts b/src/framework.ts index 470fda673..44d2b9414 100644 --- a/src/framework.ts +++ b/src/framework.ts @@ -730,7 +730,13 @@ function logSeedOmissions( * left the queue: it logs "Failed to compress chunk (hierarchical)" with * this error and records the call, and the chunk, still uncompressed, is * queued again at that manager's next rebuild (its next initialization, - * message, sent compile or released hold). + * message, sent compile or released hold). That holds where the withdrawal + * leaves the tick: a merge's call, or a chunk's first. One inside + * context-manager 0.14's compression fallback ladder, after a chunk's first + * call was refused or cut short, is caught by the rung that made it and + * recorded as a provider error, since 0.14 rethrows only its own + * CompressionBranchDiscard there; once every rung has failed, the chunk's + * request family is quarantined, with its alert, and the tick returns. */ class MaintenanceWithdrawnError extends Error { constructor(reason: string) { @@ -739,6 +745,20 @@ class MaintenanceWithdrawnError extends Error { } } +/** + * A runtime-settings preview's reason, from what stopped it: the strategy's + * known throws as the codes `RuntimeSettingsPreview.reason` documents, and + * anything else verbatim. Message matching is fragile by nature, so an + * unknown message surfaces as the reason rather than being swallowed. + */ +function previewStoppedReason(error: unknown): string { + const message = error instanceof Error ? error.message : String(error); + if (message.includes('requires reinitialization for the current branch generation')) return 'branch_generation_changed'; + if (message.includes('already running; previews must not overlap')) return 'preview_in_flight'; + if (message.includes('requires adaptiveResolution')) return 'no_adaptive_resolution'; + return message; +} + class DiscordAwarenessAccountingError extends Error { constructor(operation: string, cause: unknown) { const detail = cause instanceof Error ? cause.message : String(cause); @@ -1562,6 +1582,8 @@ export class AgentFramework { private maintenanceFailureReported = new Map(); /** Run records whose failure repeated one already reported, and whose report was suppressed. */ private maintenanceRepeats = new WeakSet(); + /** An operator's maintain (maintenanceTick) asked for the pass running or about to run: it's kept in the history whatever it met. */ + private maintenancePassRequested = false; private providerGates: Map = new Map(); private providerAccelerationCooldowns: Map = new Map(); private providerAccelerationRecoveries: Map = new Map(); @@ -3559,8 +3581,11 @@ export class AgentFramework { this.currentMaintenanceRun = null; // A pass that only met failures already reported, and took no tick, // adds nothing the history doesn't hold, and a lasting failure would - // otherwise fill it in minutes, pushing out the runs that did work. - if (!run.agents.every((record) => record.ticks === 0 && this.maintenanceRepeats.has(record))) this.maintenanceHistory.unshift(run); + // otherwise fill it in minutes, pushing out the runs that did work. One + // an operator asked for is kept: it's the pass their maintain reports. + const requested = this.maintenancePassRequested; + this.maintenancePassRequested = false; + if (requested || !run.agents.every((record) => record.ticks === 0 && this.maintenanceRepeats.has(record))) this.maintenanceHistory.unshift(run); if (this.maintenanceHistory.length > 50) this.maintenanceHistory.length = 50; } } @@ -5121,8 +5146,11 @@ export class AgentFramework { ); return { ...this.getContextMaintenanceSnapshot(), ran: false }; } + this.maintenancePassRequested = true; this.startQueuedMaintenance(); await this.maintenancePass; + // A pass with nothing queued records no run, and the next pass wasn't asked for. + this.maintenancePassRequested = false; return { ...this.getContextMaintenanceSnapshot(), ran: true }; } @@ -5356,7 +5384,7 @@ export class AgentFramework { // its budget and the patch's path, and whatever stopped the preview is // its reason. When the manager answers, anything else throws as before. if (agent.contextManagerAnswers()) throw error; - return { available: false, reason: error instanceof Error ? error.message : String(error), ...agent.runtimeBudgetPath(patch) }; + return { available: false, reason: previewStoppedReason(error), ...agent.runtimeBudgetPath(patch) }; } const base: RuntimeSettingsPreview = { available: false, @@ -5381,20 +5409,7 @@ export class AgentFramework { } return { result: result as Record }; } catch (error) { - const message = error instanceof Error ? error.message : String(error); - // Structured mapping of the strategy's known throw paths. Message - // matching is fragile by nature, so unknown messages surface verbatim - // as the reason rather than being swallowed or rethrown. - if (message.includes('requires reinitialization for the current branch generation')) { - return { reason: 'branch_generation_changed' }; - } - if (message.includes('already running; previews must not overlap')) { - return { reason: 'preview_in_flight' }; - } - if (message.includes('requires adaptiveResolution')) { - return { reason: 'no_adaptive_resolution' }; - } - return { reason: message }; + return { reason: previewStoppedReason(error) }; } }; @@ -15793,12 +15808,16 @@ export class AgentFramework { { skipLog: true }, ); } - if (agent && overBudget && drainState === 'kicking') { + // A stop() or a surgery begun from inside the alert, by an ops:alert + // listener, is read again here. + if (agent && overBudget && drainState === 'kicking' && !this.providerAdmissionClosed && !this.surgeryHold) { this.overBudgetDrainInFlight.add(agentName); // Like a maintenance pass: none starts once stop() has begun, it starts // no tick while a surgery holds the store, and a branch move, or stop(), - // waits for the tick in flight. - const drain = (async () => { + // waits for the tick in flight. It joins the drains they wait for + // before its body runs, so a stop() made from inside its first tick, + // before that tick's first await, waits for it too. + const drain = Promise.resolve().then(async () => { let ticks = 0; try { const cm = agent.getContextManager(); @@ -15817,7 +15836,7 @@ export class AgentFramework { } finally { this.overBudgetDrainInFlight.delete(agentName); } - })(); + }); this.overBudgetDrains.add(drain); void drain.finally(() => { this.overBudgetDrains.delete(drain); }); } diff --git a/test/host-quiesce.test.ts b/test/host-quiesce.test.ts index c260dd301..aa6a1978b 100644 --- a/test/host-quiesce.test.ts +++ b/test/host-quiesce.test.ts @@ -382,6 +382,31 @@ test('resume proceeds without a verdict for an agent whose context manager refus }); }); +test('a preview stopped by a stale branch generation says branch_generation_changed, as the reason documents', async () => { + const membrane = new MockMembrane(); + await withFramework(membrane, async (framework) => { + await framework.quiesce(); + // As the autobiographical strategy refuses a branch it hasn't reloaded: + // its readiness and its settings both say so. + const stale = 'TestStrategy.getHotContextSettings requires reinitialization for the current branch generation'; + const strategy = { + name: 'stale-generation-test', + async initialize() {}, + checkReadiness: () => { throw new Error(stale); }, + getHotContextSettings: () => { throw new Error(stale); }, + updateHotContextSettings: () => { throw new Error(stale); }, + async tick() {}, + select: () => [], + } as unknown as ContextStrategy; + const cm = framework.getAgent('agent')!.getContextManager(); + await cm.setStrategy(strategy); + const preview = framework.previewAgentRuntimeSettings('agent', { contextBudgetTokens: 50_000 }); + assert.equal(preview.available, false); + assert.equal(preview.reason, 'branch_generation_changed'); + assert.equal(preview.liveBudgetTokens, 100_000); + }); +}); + test('maintenanceTick runs the real maintenance pass while quiesced', async () => { class QueuedStrategy implements ContextStrategy { readonly name = 'queued-test'; diff --git a/test/maintenance-timer.test.ts b/test/maintenance-timer.test.ts index 650ffc677..1b0a0da3d 100644 --- a/test/maintenance-timer.test.ts +++ b/test/maintenance-timer.test.ts @@ -363,6 +363,35 @@ function heldFor(ms: number): MembraneError { }); } +/** A strategy whose first tick calls stop() before its first await, then notes whether the store was still open when it ended. */ +class StopInTickStrategy implements ContextStrategy { + readonly name = 'stop-in-tick-test'; + ticks = 0; + framework?: AgentFramework; + stopping?: Promise; + storeOpenAtTickEnd?: boolean; + + checkReadiness(): ReadinessState { + return { ready: false, description: 'compression queued' }; + } + + async tick(_ctx: StrategyContext): Promise { + this.ticks++; + if (this.ticks > 1) return; + this.stopping = this.framework!.stop(); + await new Promise((resolve) => setTimeout(resolve, 50)); + this.storeOpenAtTickEnd = !(this.framework as unknown as { store: { isClosed(): boolean } }).store.isClosed(); + } + + select( + _store: MessageStoreView, + _log: ContextLogView, + _budget: TokenBudget, + ): ContextEntry[] { + return []; + } +} + /** A strategy whose first tick waits on a gate, then calls its manager's membrane, as a compression would. */ class CallAfterGateStrategy implements ContextStrategy { readonly name = 'call-after-gate-test'; @@ -1338,6 +1367,36 @@ describe('queued context maintenance timer', () => { } }); + it("an operator's maintain keeps the pass it ran in the history, though the pass only met a failure already reported", { timeout: 10_000 }, async () => { + const dir = mkdtempSync(join(tmpdir(), 'af-maintenance-requested-pass-')); + const framework = await AgentFramework.create({ + storePath: join(dir, 'store'), + membrane, + agents: [{ name: 'agent', model: 'test-model', systemPrompt: 'test', strategy: new FailingStrategy() }], + modules: [], + syncIntervalMs: 0, + maintenanceIntervalMs: 0, + }); + (framework as any).opsAlert = () => {}; + const quiet = console.error; + console.error = () => {}; + try { + framework.start(); + const first = await framework.maintenanceTick(); + assert.equal(first.ran, true); + const reported = first.history[0]!.id; + assert.ok(first.history[0]!.agents[0]!.error, 'the first pass reports the failure'); + const second = await framework.maintenanceTick(); + assert.equal(second.ran, true); + assert.ok(second.history[0]!.id > reported, 'the newest run is the pass the second maintain ran'); + assert.ok(second.history[0]!.agents[0]!.error); + } finally { + console.error = quiet; + await framework.stop(); + rmSync(dir, { recursive: true, force: true }); + } + }); + it('a pass that took a tick before meeting a failure already reported stays in the history', { timeout: 10_000 }, async () => { const dir = mkdtempSync(join(tmpdir(), 'af-maintenance-repeats-after-work-')); const framework = await AgentFramework.create({ @@ -1483,6 +1542,67 @@ describe('queued context maintenance timer', () => { } }); + it("a stop() begun from inside the refusal's alert starts no drain kick", async () => { + const dir = mkdtempSync(join(tmpdir(), 'af-maintenance-drain-stop-in-alert-')); + const framework = await AgentFramework.create({ + storePath: join(dir, 'store'), + membrane, + agents: [{ name: 'agent', model: 'test-model', systemPrompt: 'test', strategy: new ReadyStrategy() }], + modules: [], + syncIntervalMs: 0, + maintenanceIntervalMs: 0, + }); + const internals = framework as unknown as { + noteInferenceExhausted(agentName: string, reason: string, retryable?: boolean, errorType?: string): void; + overBudgetDrains: Set>; + }; + let stopping: Promise | undefined; + // An ops:alert listener that stops the framework, as the alert is raised. + (framework as any).opsAlert = () => { stopping ??= framework.stop(); }; + const quiet = console.error; + console.error = () => {}; + try { + internals.noteInferenceExhausted('agent', 'compile refused', false, 'over_budget'); + assert.ok(stopping, 'the alert began stop()'); + assert.equal(internals.overBudgetDrains.size, 0, 'no kick was started'); + await stopping; + } finally { + console.error = quiet; + await (stopping ?? framework.stop()); + rmSync(dir, { recursive: true, force: true }); + } + }); + + it("a stop() made from a drain's first tick, before that tick's first await, waits for the tick", { timeout: 10_000 }, async () => { + const dir = mkdtempSync(join(tmpdir(), 'af-maintenance-drain-stop-in-tick-')); + const strategy = new StopInTickStrategy(); + const framework = await AgentFramework.create({ + storePath: join(dir, 'store'), + membrane, + agents: [{ name: 'agent', model: 'test-model', systemPrompt: 'test', strategy }], + modules: [], + syncIntervalMs: 0, + maintenanceIntervalMs: 0, + }); + strategy.framework = framework; + const internals = framework as unknown as { + noteInferenceExhausted(agentName: string, reason: string, retryable?: boolean, errorType?: string): void; + }; + (framework as any).opsAlert = () => {}; + const quiet = console.error; + console.error = () => {}; + try { + internals.noteInferenceExhausted('agent', 'compile refused', false, 'over_budget'); + await waitFor(() => strategy.stopping !== undefined); + await strategy.stopping; + assert.equal(strategy.storeOpenAtTickEnd, true, 'the tick ended before the store closed'); + } finally { + console.error = quiet; + await (strategy.stopping ?? framework.stop()); + rmSync(dir, { recursive: true, force: true }); + } + }); + it('no over-budget drain kick starts once stop() has begun', async () => { const dir = mkdtempSync(join(tmpdir(), 'af-maintenance-drain-after-stop-')); const framework = await AgentFramework.create({ @@ -1580,8 +1700,6 @@ describe('queued context maintenance timer', () => { const quiet = console.error; console.error = () => {}; try { - // A drain kick ticks only while the framework runs. - framework.start(); // A refused compile kicks the drain, whose first tick waits on the gate. internals.noteInferenceExhausted('agent', 'compile refused', false, 'over_budget'); await waitFor(() => strategy.ticks === 1); From e34d8020849e46c1e5b7a767149ee29e7c49705a Mon Sep 17 00:00:00 2001 From: Gladys-2413 Date: Thu, 15 Oct 2026 23:30:37 -0400 Subject: [PATCH 07/11] fix(maintenance): pin the maintain request's reset, and scope the withdrawal text to the fallback ladder Nothing pinned the reset of the maintain request after maintenanceTick()'s await. Without it, a maintain whose pass had nothing queued (the agent's own call holding its provider gate) left the request set, and the next pass, which nobody asked for and which met only the failure already reported, was kept in the history. A test now runs that case. The withdrawal's doc and the changelog said context-manager counts no attempt for a merge's call or a chunk's first. A chunk's canonical call's retries after a tool_use stop are outside the compression fallback ladder too, and their throws reach the same catch, which logs "Failed to compress chunk (hierarchical)" and rethrows; the text now says it holds for every call outside the ladder. The changelog no longer says "until context-manager rethrows such a withdrawal", which would promise context-manager's work: "context-manager 0.14's" scopes the sentence. --- changelog.d/cm-0.14.changed.md | 2 +- src/framework.ts | 15 ++++++------ test/maintenance-timer.test.ts | 45 ++++++++++++++++++++++++++++++++++ 3 files changed, 54 insertions(+), 8 deletions(-) diff --git a/changelog.d/cm-0.14.changed.md b/changelog.d/cm-0.14.changed.md index 6dea6b12e..625060680 100644 --- a/changelog.d/cm-0.14.changed.md +++ b/changelog.d/cm-0.14.changed.md @@ -1,2 +1,2 @@ - A context-maintenance pass no longer fails as a whole when one agent's readiness can't be read. context-manager 0.14, which agent-framework has required since 0.21.1, refuses its readiness queries while a strategy's initialization is pending (inside `switchBranch`, `fork`, `setStrategy` or `ContextManager.open`) or has failed, and a strategy can refuse on its own, as the autobiographical one does for a branch it hasn't reloaded. That agent's run now records the error, the agent gets its own `context-maintenance-failed` alert (`scope: 'agent'`), and the other agents run. Before, such a throw ended the pass for every agent, with one alert for the whole pass (`scope: 'pass'`). Anything else that throws while one agent's maintenance is prepared is that agent's failure in the same way. A failure that lasts, met by every pass, is logged, alerted and recorded in the maintenance history when it's first met, then again when its text changes, every 15 minutes while it lasts (the alert webhook's cooldown for it, counted from the webhook's last post of it), or after that agent's maintenance has gone without a failure. A pass that met only failures already reported, and took no tick, isn't added to the history unless an operator's `maintain` (`maintenanceTick()`) asked for it, so a lasting failure no longer pushes out the runs that did work. `getContextMaintenanceSnapshot()` reports such an agent as `ready: false` with an `error`, and a strategy that can't give a progress snapshot reads as having none. -- While an operator surgery (`rollbackToMessage` or `suppressMessages`) or `switchBranch` holds the store, no maintenance pass starts and no over-budget drain kick is started, a running pass or drain kick starts no new tick, and the operator call waits for them to finish before it switches or forks the branch, so no maintenance work overlaps the move. `stop()` waits for them the same way, a drain kick included, before it closes the store, so the tick in flight ends before the store closes, and once `stop()` has begun, no drain kick starts and none starts a new tick. Maintenance makes no provider call meanwhile, a conversation agent's included: one made under the hold, or once `stop()` has begun, is withdrawn without being sent, and so is a resident's call waiting at its provider gate when the hold is taken or `stop()` begins. That's no maintenance failure, and context-manager counts no attempt for it, where the withdrawal is a merge's call or a chunk's first. One inside context-manager 0.14's compression fallback ladder, after a chunk's first call was refused or cut short, is recorded there as a provider error, and once every rung has failed, the chunk is quarantined, with its alert, until context-manager rethrows such a withdrawal. Before, a resident's call made once `stop()` had begun was refused as a failure, so a restart that caught a maintenance tick before one of its calls logged a maintenance failure and raised a `context-maintenance-failed` alert; now the agent's maintenance logs that it stopped. An over-budget compile's `context-refusal` alert says whether the drain breaker kicked: `kicking`, `already draining`, or `not kicked` with the reason, where it said `kicking` whenever the agent was found. A merge keeps its place in the queue. A chunk's compression logs context-manager's "Failed to compress chunk (hierarchical)" line with the withdrawal, and the chunk is queued again at that manager's next rebuild, such as its next message or compile. The operator call never waits on a provider cooldown, or on the retry turn a cooldown queues, which the hold keeps from starting. `switchBranch` can therefore move the store after the call returns, and the API server's `branch.switch` announces the move it made once it has, while `branch.current`, the branch list, `/health` and `/status` read the store's branch whenever they report one. `maintenanceTick()` answers `ran: false` meanwhile, and `resume()` refuses with `OperatorActionError` (`agent-busy`), as the other operator calls do, including when the surgery took the store while `resume()` waited. Resume once the surgery has finished. +- While an operator surgery (`rollbackToMessage` or `suppressMessages`) or `switchBranch` holds the store, no maintenance pass starts and no over-budget drain kick is started, a running pass or drain kick starts no new tick, and the operator call waits for them to finish before it switches or forks the branch, so no maintenance work overlaps the move. `stop()` waits for them the same way, a drain kick included, before it closes the store, so the tick in flight ends before the store closes, and once `stop()` has begun, no drain kick starts and none starts a new tick. Maintenance makes no provider call meanwhile, a conversation agent's included: one made under the hold, or once `stop()` has begun, is withdrawn without being sent, and so is a resident's call waiting at its provider gate when the hold is taken or `stop()` begins. That's no maintenance failure, and for a call outside context-manager 0.14's compression fallback ladder, a merge's or a chunk's canonical call with its retries after a `tool_use` stop, context-manager counts no attempt. One inside the ladder, after a chunk's canonical call was refused or cut short, is recorded there as a provider error, and once every rung has failed, the chunk is quarantined, with its alert. Before, a resident's call made once `stop()` had begun was refused as a failure, so a restart that caught a maintenance tick before one of its calls logged a maintenance failure and raised a `context-maintenance-failed` alert; now the agent's maintenance logs that it stopped. An over-budget compile's `context-refusal` alert says whether the drain breaker kicked: `kicking`, `already draining`, or `not kicked` with the reason, where it said `kicking` whenever the agent was found. A merge keeps its place in the queue. A chunk's compression logs context-manager's "Failed to compress chunk (hierarchical)" line with the withdrawal, and the chunk is queued again at that manager's next rebuild, such as its next message or compile. The operator call never waits on a provider cooldown, or on the retry turn a cooldown queues, which the hold keeps from starting. `switchBranch` can therefore move the store after the call returns, and the API server's `branch.switch` announces the move it made once it has, while `branch.current`, the branch list, `/health` and `/status` read the store's branch whenever they report one. `maintenanceTick()` answers `ran: false` meanwhile, and `resume()` refuses with `OperatorActionError` (`agent-busy`), as the other operator calls do, including when the surgery took the store while `resume()` waited. Resume once the surgery has finished. diff --git a/src/framework.ts b/src/framework.ts index 44d2b9414..10b3ba236 100644 --- a/src/framework.ts +++ b/src/framework.ts @@ -730,13 +730,14 @@ function logSeedOmissions( * left the queue: it logs "Failed to compress chunk (hierarchical)" with * this error and records the call, and the chunk, still uncompressed, is * queued again at that manager's next rebuild (its next initialization, - * message, sent compile or released hold). That holds where the withdrawal - * leaves the tick: a merge's call, or a chunk's first. One inside - * context-manager 0.14's compression fallback ladder, after a chunk's first - * call was refused or cut short, is caught by the rung that made it and - * recorded as a provider error, since 0.14 rethrows only its own - * CompressionBranchDiscard there; once every rung has failed, the chunk's - * request family is quarantined, with its alert, and the tick returns. + * message, sent compile or released hold). That holds for every call outside + * context-manager 0.14's compression fallback ladder: a merge's, and a + * chunk's canonical call with its retries after a tool_use stop. One inside + * the ladder, after a chunk's canonical call was refused or cut short, is + * caught by the rung that made it and recorded as a provider error, since + * 0.14 rethrows only its own CompressionBranchDiscard there; once every rung + * has failed, the chunk's request family is quarantined, with its alert, and + * the tick returns. */ class MaintenanceWithdrawnError extends Error { constructor(reason: string) { diff --git a/test/maintenance-timer.test.ts b/test/maintenance-timer.test.ts index 1b0a0da3d..a60951a6f 100644 --- a/test/maintenance-timer.test.ts +++ b/test/maintenance-timer.test.ts @@ -1397,6 +1397,51 @@ describe('queued context maintenance timer', () => { } }); + + it("a maintain whose pass had nothing to run leaves no request for the next pass", { timeout: 10_000 }, async () => { + // Gordon-2414's probe. While the agent's own call holds its provider gate, + // a maintain's pass queues nothing and records no run, so the next pass, + // which nobody asked for and which meets only the failure already + // reported, stays out of the history. + const dir = mkdtempSync(join(tmpdir(), 'af-maintenance-request-reset-')); + const framework = await AgentFramework.create({ + storePath: join(dir, 'store'), + membrane, + agents: [{ name: 'agent', model: 'test-model', systemPrompt: 'test', strategy: new FailingStrategy() }], + modules: [], + syncIntervalMs: 0, + maintenanceIntervalMs: 0, + }); + const internals = framework as unknown as { + acquirePrimaryProviderGate(agentName: string): void; + releasePrimaryProviderGate(agentName: string): void; + startQueuedMaintenance(): void; + maintenancePass: Promise | null; + }; + (framework as any).opsAlert = () => {}; + const quiet = console.error; + const quietWarn = console.warn; + console.error = () => {}; + console.warn = () => {}; + try { + framework.start(); + const first = await framework.maintenanceTick(); + assert.equal(first.history.length, 1, 'the first maintain reports the failure'); + internals.acquirePrimaryProviderGate('agent'); + const second = await framework.maintenanceTick(); + assert.equal(second.ran, true); + assert.equal(second.history.length, 1, 'a pass with nothing queued records no run'); + internals.releasePrimaryProviderGate('agent'); + internals.startQueuedMaintenance(); + await internals.maintenancePass; + assert.equal(framework.getContextMaintenanceSnapshot().history.length, 1, 'the repeat nobody asked for stays out of the history'); + } finally { + console.error = quiet; + console.warn = quietWarn; + await framework.stop(); + rmSync(dir, { recursive: true, force: true }); + } + }); it('a pass that took a tick before meeting a failure already reported stays in the history', { timeout: 10_000 }, async () => { const dir = mkdtempSync(join(tmpdir(), 'af-maintenance-repeats-after-work-')); const framework = await AgentFramework.create({ From ef752662a04a8a7744d22bb4cb46e38c93d11369 Mon Sep 17 00:00:00 2001 From: Gladys-2413 Date: Thu, 15 Oct 2026 23:37:40 -0400 Subject: [PATCH 08/11] docs(changelog): a chunk is queued again only when withdrawn outside the fallback ladder The fragment's later sentence said a chunk whose compression was withdrawn logs context-manager's "Failed to compress chunk (hierarchical)" line and is queued again at the manager's next rebuild. That holds outside context-manager 0.14's compression fallback ladder; inside it, the withdrawal is recorded as a provider error and the chunk can be quarantined, as the fragment's earlier sentence now says. The sentence is now scoped the same way. Changelog text only. --- changelog.d/cm-0.14.changed.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/changelog.d/cm-0.14.changed.md b/changelog.d/cm-0.14.changed.md index 625060680..9d55c3d06 100644 --- a/changelog.d/cm-0.14.changed.md +++ b/changelog.d/cm-0.14.changed.md @@ -1,2 +1,2 @@ - A context-maintenance pass no longer fails as a whole when one agent's readiness can't be read. context-manager 0.14, which agent-framework has required since 0.21.1, refuses its readiness queries while a strategy's initialization is pending (inside `switchBranch`, `fork`, `setStrategy` or `ContextManager.open`) or has failed, and a strategy can refuse on its own, as the autobiographical one does for a branch it hasn't reloaded. That agent's run now records the error, the agent gets its own `context-maintenance-failed` alert (`scope: 'agent'`), and the other agents run. Before, such a throw ended the pass for every agent, with one alert for the whole pass (`scope: 'pass'`). Anything else that throws while one agent's maintenance is prepared is that agent's failure in the same way. A failure that lasts, met by every pass, is logged, alerted and recorded in the maintenance history when it's first met, then again when its text changes, every 15 minutes while it lasts (the alert webhook's cooldown for it, counted from the webhook's last post of it), or after that agent's maintenance has gone without a failure. A pass that met only failures already reported, and took no tick, isn't added to the history unless an operator's `maintain` (`maintenanceTick()`) asked for it, so a lasting failure no longer pushes out the runs that did work. `getContextMaintenanceSnapshot()` reports such an agent as `ready: false` with an `error`, and a strategy that can't give a progress snapshot reads as having none. -- While an operator surgery (`rollbackToMessage` or `suppressMessages`) or `switchBranch` holds the store, no maintenance pass starts and no over-budget drain kick is started, a running pass or drain kick starts no new tick, and the operator call waits for them to finish before it switches or forks the branch, so no maintenance work overlaps the move. `stop()` waits for them the same way, a drain kick included, before it closes the store, so the tick in flight ends before the store closes, and once `stop()` has begun, no drain kick starts and none starts a new tick. Maintenance makes no provider call meanwhile, a conversation agent's included: one made under the hold, or once `stop()` has begun, is withdrawn without being sent, and so is a resident's call waiting at its provider gate when the hold is taken or `stop()` begins. That's no maintenance failure, and for a call outside context-manager 0.14's compression fallback ladder, a merge's or a chunk's canonical call with its retries after a `tool_use` stop, context-manager counts no attempt. One inside the ladder, after a chunk's canonical call was refused or cut short, is recorded there as a provider error, and once every rung has failed, the chunk is quarantined, with its alert. Before, a resident's call made once `stop()` had begun was refused as a failure, so a restart that caught a maintenance tick before one of its calls logged a maintenance failure and raised a `context-maintenance-failed` alert; now the agent's maintenance logs that it stopped. An over-budget compile's `context-refusal` alert says whether the drain breaker kicked: `kicking`, `already draining`, or `not kicked` with the reason, where it said `kicking` whenever the agent was found. A merge keeps its place in the queue. A chunk's compression logs context-manager's "Failed to compress chunk (hierarchical)" line with the withdrawal, and the chunk is queued again at that manager's next rebuild, such as its next message or compile. The operator call never waits on a provider cooldown, or on the retry turn a cooldown queues, which the hold keeps from starting. `switchBranch` can therefore move the store after the call returns, and the API server's `branch.switch` announces the move it made once it has, while `branch.current`, the branch list, `/health` and `/status` read the store's branch whenever they report one. `maintenanceTick()` answers `ran: false` meanwhile, and `resume()` refuses with `OperatorActionError` (`agent-busy`), as the other operator calls do, including when the surgery took the store while `resume()` waited. Resume once the surgery has finished. +- While an operator surgery (`rollbackToMessage` or `suppressMessages`) or `switchBranch` holds the store, no maintenance pass starts and no over-budget drain kick is started, a running pass or drain kick starts no new tick, and the operator call waits for them to finish before it switches or forks the branch, so no maintenance work overlaps the move. `stop()` waits for them the same way, a drain kick included, before it closes the store, so the tick in flight ends before the store closes, and once `stop()` has begun, no drain kick starts and none starts a new tick. Maintenance makes no provider call meanwhile, a conversation agent's included: one made under the hold, or once `stop()` has begun, is withdrawn without being sent, and so is a resident's call waiting at its provider gate when the hold is taken or `stop()` begins. That's no maintenance failure, and for a call outside context-manager 0.14's compression fallback ladder, a merge's or a chunk's canonical call with its retries after a `tool_use` stop, context-manager counts no attempt. One inside the ladder, after a chunk's canonical call was refused or cut short, is recorded there as a provider error, and once every rung has failed, the chunk is quarantined, with its alert. Before, a resident's call made once `stop()` had begun was refused as a failure, so a restart that caught a maintenance tick before one of its calls logged a maintenance failure and raised a `context-maintenance-failed` alert; now the agent's maintenance logs that it stopped. An over-budget compile's `context-refusal` alert says whether the drain breaker kicked: `kicking`, `already draining`, or `not kicked` with the reason, where it said `kicking` whenever the agent was found. A merge keeps its place in the queue. When a chunk's compression is withdrawn outside the ladder, context-manager logs its "Failed to compress chunk (hierarchical)" line with the withdrawal, and the chunk is queued again at that manager's next rebuild, such as its next message or compile. The operator call never waits on a provider cooldown, or on the retry turn a cooldown queues, which the hold keeps from starting. `switchBranch` can therefore move the store after the call returns, and the API server's `branch.switch` announces the move it made once it has, while `branch.current`, the branch list, `/health` and `/status` read the store's branch whenever they report one. `maintenanceTick()` answers `ran: false` meanwhile, and `resume()` refuses with `OperatorActionError` (`agent-busy`), as the other operator calls do, including when the surgery took the store while `resume()` waited. Resume once the surgery has finished. From d82dc70931b25ce03c07637a365a28e1ef5941f3 Mon Sep 17 00:00:00 2001 From: Gladys-2413 Date: Fri, 16 Oct 2026 01:11:08 -0400 Subject: [PATCH 09/11] fix(maintenance): the drain kick joins what stop() and surgeries wait for before the refusal's alert The refusal's alert was raised before the kick, and the kick's decision was read again after it, so an ops:alert listener that began stop() or a surgery left the alert saying "drain breaker kicking" while no kick started, and a refusal raised again from inside a listener could start a second drain for the same agent: the reread didn't look at the in-flight flag. The kick now joins the drains, with its in-flight flag set, before the alert is raised, and the reread is gone. The alert's "kicking" is then the kick's own decision: a listener that begins stop() or a surgery waits for the kick, whose body ticks none, since its loop reads admission and the hold before each tick, and logs that it stopped and why. A refusal raised again from a listener finds the flag and says "already draining". Tests: the alert-listener stop() test now asserts the alert's "kicking", that the kick had joined the drains when the alert was raised, that it ticked none, and its log line; a surgery begun from the alert's listener does the same. The stop-from-first-tick test races stop() against a second instead of reading the store after a fixed 50 ms, and asserts stop() hadn't settled when the tick ended. A maintain that joins a pass already running keeps that pass in the history. The nothing-to-run test counts the ticks, so it shows the third pass ran. Text: the withdrawal's doc puts its scope first, so the merge's and the chunk's accounts read as the calls outside the fallback ladder; the changelog names the quarantine as the chunk's request family's, lists the alert's "unavailable (no agent handle)", as before, and says a listener's stop() or surgery waits for the kick; the request flag's doc says a pass with nothing to run records no run; the drain's comment names what waits for its tick in flight: a surgery, switchBranch or stop(). --- changelog.d/cm-0.14.changed.md | 2 +- src/framework.ts | 64 ++++++------- test/maintenance-timer.test.ts | 167 +++++++++++++++++++++++++++++++-- 3 files changed, 193 insertions(+), 40 deletions(-) diff --git a/changelog.d/cm-0.14.changed.md b/changelog.d/cm-0.14.changed.md index 9d55c3d06..b2ec49603 100644 --- a/changelog.d/cm-0.14.changed.md +++ b/changelog.d/cm-0.14.changed.md @@ -1,2 +1,2 @@ - A context-maintenance pass no longer fails as a whole when one agent's readiness can't be read. context-manager 0.14, which agent-framework has required since 0.21.1, refuses its readiness queries while a strategy's initialization is pending (inside `switchBranch`, `fork`, `setStrategy` or `ContextManager.open`) or has failed, and a strategy can refuse on its own, as the autobiographical one does for a branch it hasn't reloaded. That agent's run now records the error, the agent gets its own `context-maintenance-failed` alert (`scope: 'agent'`), and the other agents run. Before, such a throw ended the pass for every agent, with one alert for the whole pass (`scope: 'pass'`). Anything else that throws while one agent's maintenance is prepared is that agent's failure in the same way. A failure that lasts, met by every pass, is logged, alerted and recorded in the maintenance history when it's first met, then again when its text changes, every 15 minutes while it lasts (the alert webhook's cooldown for it, counted from the webhook's last post of it), or after that agent's maintenance has gone without a failure. A pass that met only failures already reported, and took no tick, isn't added to the history unless an operator's `maintain` (`maintenanceTick()`) asked for it, so a lasting failure no longer pushes out the runs that did work. `getContextMaintenanceSnapshot()` reports such an agent as `ready: false` with an `error`, and a strategy that can't give a progress snapshot reads as having none. -- While an operator surgery (`rollbackToMessage` or `suppressMessages`) or `switchBranch` holds the store, no maintenance pass starts and no over-budget drain kick is started, a running pass or drain kick starts no new tick, and the operator call waits for them to finish before it switches or forks the branch, so no maintenance work overlaps the move. `stop()` waits for them the same way, a drain kick included, before it closes the store, so the tick in flight ends before the store closes, and once `stop()` has begun, no drain kick starts and none starts a new tick. Maintenance makes no provider call meanwhile, a conversation agent's included: one made under the hold, or once `stop()` has begun, is withdrawn without being sent, and so is a resident's call waiting at its provider gate when the hold is taken or `stop()` begins. That's no maintenance failure, and for a call outside context-manager 0.14's compression fallback ladder, a merge's or a chunk's canonical call with its retries after a `tool_use` stop, context-manager counts no attempt. One inside the ladder, after a chunk's canonical call was refused or cut short, is recorded there as a provider error, and once every rung has failed, the chunk is quarantined, with its alert. Before, a resident's call made once `stop()` had begun was refused as a failure, so a restart that caught a maintenance tick before one of its calls logged a maintenance failure and raised a `context-maintenance-failed` alert; now the agent's maintenance logs that it stopped. An over-budget compile's `context-refusal` alert says whether the drain breaker kicked: `kicking`, `already draining`, or `not kicked` with the reason, where it said `kicking` whenever the agent was found. A merge keeps its place in the queue. When a chunk's compression is withdrawn outside the ladder, context-manager logs its "Failed to compress chunk (hierarchical)" line with the withdrawal, and the chunk is queued again at that manager's next rebuild, such as its next message or compile. The operator call never waits on a provider cooldown, or on the retry turn a cooldown queues, which the hold keeps from starting. `switchBranch` can therefore move the store after the call returns, and the API server's `branch.switch` announces the move it made once it has, while `branch.current`, the branch list, `/health` and `/status` read the store's branch whenever they report one. `maintenanceTick()` answers `ran: false` meanwhile, and `resume()` refuses with `OperatorActionError` (`agent-busy`), as the other operator calls do, including when the surgery took the store while `resume()` waited. Resume once the surgery has finished. +- While an operator surgery (`rollbackToMessage` or `suppressMessages`) or `switchBranch` holds the store, no maintenance pass starts and no over-budget drain kick is started, a running pass or drain kick starts no new tick, and the operator call waits for them to finish before it switches or forks the branch, so no maintenance work overlaps the move. `stop()` waits for them the same way, a drain kick included, before it closes the store, so the tick in flight ends before the store closes, and once `stop()` has begun, no drain kick starts and none starts a new tick. Maintenance makes no provider call meanwhile, a conversation agent's included: one made under the hold, or once `stop()` has begun, is withdrawn without being sent, and so is a resident's call waiting at its provider gate when the hold is taken or `stop()` begins. That's no maintenance failure, and for a call outside context-manager 0.14's compression fallback ladder, a merge's or a chunk's canonical call with its retries after a `tool_use` stop, context-manager counts no attempt. One inside the ladder, after a chunk's canonical call was refused or cut short, is recorded there as a provider error, and once every rung has failed, the chunk's request family is quarantined, with its alert. Before, a resident's call made once `stop()` had begun was refused as a failure, so a restart that caught a maintenance tick before one of its calls logged a maintenance failure and raised a `context-maintenance-failed` alert; now the agent's maintenance logs that it stopped. An over-budget compile's `context-refusal` alert says whether the drain breaker kicked: `kicking`, `already draining`, or `not kicked` with the reason, where it said `kicking` whenever the agent was found, and `unavailable (no agent handle)` for one it can't find, as before. A kick joins what `stop()` and an operator surgery wait for before the alert is raised, so a listener on the alert that begins either waits for the kick, which then starts no tick. A merge keeps its place in the queue. When a chunk's compression is withdrawn outside the ladder, context-manager logs its "Failed to compress chunk (hierarchical)" line with the withdrawal, and the chunk is queued again at that manager's next rebuild, such as its next message or compile. The operator call never waits on a provider cooldown, or on the retry turn a cooldown queues, which the hold keeps from starting. `switchBranch` can therefore move the store after the call returns, and the API server's `branch.switch` announces the move it made once it has, while `branch.current`, the branch list, `/health` and `/status` read the store's branch whenever they report one. `maintenanceTick()` answers `ran: false` meanwhile, and `resume()` refuses with `OperatorActionError` (`agent-busy`), as the other operator calls do, including when the surgery took the store while `resume()` waited. Resume once the surgery has finished. diff --git a/src/framework.ts b/src/framework.ts index 10b3ba236..d4bce939c 100644 --- a/src/framework.ts +++ b/src/framework.ts @@ -724,20 +724,19 @@ function logSeedOmissions( * A maintenance provider call withdrawn because a surgery holds the store, or * because stop() has closed provider admission: either is the framework's own * doing, and no failure. It sent nothing: it was refused at the provider - * gate, or woken there from its wait. context-manager counts no attempt for - * it. A merge keeps its queue entry, so the next tick after the hold, or - * after the next start, does it again. A chunk's compression has already - * left the queue: it logs "Failed to compress chunk (hierarchical)" with - * this error and records the call, and the chunk, still uncompressed, is - * queued again at that manager's next rebuild (its next initialization, - * message, sent compile or released hold). That holds for every call outside - * context-manager 0.14's compression fallback ladder: a merge's, and a - * chunk's canonical call with its retries after a tool_use stop. One inside - * the ladder, after a chunk's canonical call was refused or cut short, is - * caught by the rung that made it and recorded as a provider error, since - * 0.14 rethrows only its own CompressionBranchDiscard there; once every rung - * has failed, the chunk's request family is quarantined, with its alert, and - * the tick returns. + * gate, or woken there from its wait. For a call outside context-manager + * 0.14's compression fallback ladder, a merge's or a chunk's canonical call + * with its retries after a tool_use stop, context-manager counts no attempt. + * A merge keeps its queue entry, so the next tick after the hold, or after + * the next start, does it again. A chunk's compression has already left the + * queue: it logs "Failed to compress chunk (hierarchical)" with this error + * and records the call, and the chunk, still uncompressed, is queued again at + * that manager's next rebuild (its next initialization, message, sent compile + * or released hold). A call inside the ladder, after a chunk's canonical call + * was refused or cut short, is caught by the rung that made it and recorded + * as a provider error, since 0.14 rethrows only its own + * CompressionBranchDiscard there; once every rung has failed, the chunk's + * request family is quarantined, with its alert, and the tick returns. */ class MaintenanceWithdrawnError extends Error { constructor(reason: string) { @@ -1583,7 +1582,7 @@ export class AgentFramework { private maintenanceFailureReported = new Map(); /** Run records whose failure repeated one already reported, and whose report was suppressed. */ private maintenanceRepeats = new WeakSet(); - /** An operator's maintain (maintenanceTick) asked for the pass running or about to run: it's kept in the history whatever it met. */ + /** An operator's maintain (maintenanceTick) asked for the pass running or about to run: the run it records is kept in the history whatever it met. A pass with nothing to run records none. */ private maintenancePassRequested = false; private providerGates: Map = new Map(); private providerAccelerationCooldowns: Map = new Map(); @@ -15799,25 +15798,16 @@ export class AgentFramework { : this.surgeryHold ? 'not kicked: a surgery holds the store' : this.overBudgetDrainInFlight.has(agentName) ? 'already draining' : 'kicking'; - if (overBudget) { - this.opsAlert( - 'context-refusal', - agentName, - `compile refused (${errorType ?? 'unclassified'}): ${boundFailureText(wholeReason, FAILURE_SUMMARY_MAX_CHARS)} — ` + - `drain breaker ${drainState}; ` + - `if this persists, raise contextBudgetTokens or overBudgetGraceRatio.`, - { skipLog: true }, - ); - } - // A stop() or a surgery begun from inside the alert, by an ops:alert - // listener, is read again here. - if (agent && overBudget && drainState === 'kicking' && !this.providerAdmissionClosed && !this.surgeryHold) { + if (agent && drainState === 'kicking') { this.overBudgetDrainInFlight.add(agentName); // Like a maintenance pass: none starts once stop() has begun, it starts - // no tick while a surgery holds the store, and a branch move, or stop(), - // waits for the tick in flight. It joins the drains they wait for - // before its body runs, so a stop() made from inside its first tick, - // before that tick's first await, waits for it too. + // no tick while a surgery holds the store, and a surgery, a + // switchBranch or stop() waits for the tick in flight. It joins the + // drains they wait for before the alert is raised and before its body + // runs, so the alert's listeners find it: one that begins stop() or a + // surgery waits for it, and its body then ticks none and says why. So + // does a stop() made from inside its first tick, before that tick's + // first await. const drain = Promise.resolve().then(async () => { let ticks = 0; try { @@ -15841,6 +15831,16 @@ export class AgentFramework { this.overBudgetDrains.add(drain); void drain.finally(() => { this.overBudgetDrains.delete(drain); }); } + if (overBudget) { + this.opsAlert( + 'context-refusal', + agentName, + `compile refused (${errorType ?? 'unclassified'}): ${boundFailureText(wholeReason, FAILURE_SUMMARY_MAX_CHARS)} — ` + + `drain breaker ${drainState}; ` + + `if this persists, raise contextBudgetTokens or overBudgetGraceRatio.`, + { skipLog: true }, + ); + } // (3) Hard-down escalation on repeated identical failure. if (streak >= this.inferenceFailureEscalationThreshold) { diff --git a/test/maintenance-timer.test.ts b/test/maintenance-timer.test.ts index a60951a6f..138a9b0ba 100644 --- a/test/maintenance-timer.test.ts +++ b/test/maintenance-timer.test.ts @@ -369,6 +369,7 @@ class StopInTickStrategy implements ContextStrategy { ticks = 0; framework?: AgentFramework; stopping?: Promise; + stoppedBeforeTickEnd?: boolean; storeOpenAtTickEnd?: boolean; checkReadiness(): ReadinessState { @@ -379,7 +380,13 @@ class StopInTickStrategy implements ContextStrategy { this.ticks++; if (this.ticks > 1) return; this.stopping = this.framework!.stop(); - await new Promise((resolve) => setTimeout(resolve, 50)); + // A stop() that doesn't wait for this tick settles first; one that does + // can't settle before it ends, so the second second is this tick's. + const first = await Promise.race([ + this.stopping.then(() => 'stopped' as const, () => 'stopped' as const), + new Promise<'waited'>((resolve) => setTimeout(() => resolve('waited'), 1000)), + ]); + this.stoppedBeforeTickEnd = first === 'stopped'; this.storeOpenAtTickEnd = !(this.framework as unknown as { store: { isClosed(): boolean } }).store.isClosed(); } @@ -392,6 +399,36 @@ class StopInTickStrategy implements ContextStrategy { } } +/** A strategy whose every tick meets the same failure, its second only once its gate opens. */ +class GatedRepeatFailureStrategy implements ContextStrategy { + readonly name = 'gated-repeat-failure-test'; + ticks = 0; + private open?: () => void; + private readonly gate = new Promise((resolve) => { this.open = resolve; }); + + release(): void { + this.open?.(); + } + + checkReadiness(): ReadinessState { + return { ready: false, description: 'compression blocked' }; + } + + async tick(_ctx: StrategyContext): Promise { + this.ticks++; + if (this.ticks === 2) await this.gate; + throw new Error('provider rejected maintenance request'); + } + + select( + _store: MessageStoreView, + _log: ContextLogView, + _budget: TokenBudget, + ): ContextEntry[] { + return []; + } +} + /** A strategy whose first tick waits on a gate, then calls its manager's membrane, as a compression would. */ class CallAfterGateStrategy implements ContextStrategy { readonly name = 'call-after-gate-test'; @@ -1404,10 +1441,14 @@ describe('queued context maintenance timer', () => { // which nobody asked for and which meets only the failure already // reported, stays out of the history. const dir = mkdtempSync(join(tmpdir(), 'af-maintenance-request-reset-')); + const strategy = new FailingStrategy(); + let ticks = 0; + const failingTick = strategy.tick.bind(strategy); + strategy.tick = async (ctx: StrategyContext) => { ticks++; return failingTick(ctx); }; const framework = await AgentFramework.create({ storePath: join(dir, 'store'), membrane, - agents: [{ name: 'agent', model: 'test-model', systemPrompt: 'test', strategy: new FailingStrategy() }], + agents: [{ name: 'agent', model: 'test-model', systemPrompt: 'test', strategy }], modules: [], syncIntervalMs: 0, maintenanceIntervalMs: 0, @@ -1431,9 +1472,11 @@ describe('queued context maintenance timer', () => { const second = await framework.maintenanceTick(); assert.equal(second.ran, true); assert.equal(second.history.length, 1, 'a pass with nothing queued records no run'); + assert.equal(ticks, 1, 'the second maintain ran no tick'); internals.releasePrimaryProviderGate('agent'); internals.startQueuedMaintenance(); await internals.maintenancePass; + assert.equal(ticks, 2, 'the pass nobody asked for ran its tick'); assert.equal(framework.getContextMaintenanceSnapshot().history.length, 1, 'the repeat nobody asked for stays out of the history'); } finally { console.error = quiet; @@ -1442,6 +1485,47 @@ describe('queued context maintenance timer', () => { rmSync(dir, { recursive: true, force: true }); } }); + + it("a maintain that joins a pass already running keeps that pass in the history", { timeout: 10_000 }, async () => { + const dir = mkdtempSync(join(tmpdir(), 'af-maintenance-request-join-')); + const strategy = new GatedRepeatFailureStrategy(); + const framework = await AgentFramework.create({ + storePath: join(dir, 'store'), + membrane, + agents: [{ name: 'agent', model: 'test-model', systemPrompt: 'test', strategy }], + modules: [], + syncIntervalMs: 0, + maintenanceIntervalMs: 0, + }); + const internals = framework as unknown as { startQueuedMaintenance(): void }; + (framework as any).opsAlert = () => {}; + const quiet = console.error; + const quietWarn = console.warn; + console.error = () => {}; + console.warn = () => {}; + try { + framework.start(); + const first = await framework.maintenanceTick(); + assert.equal(first.history.length, 1, 'the first maintain reports the failure'); + // A pass nobody asked for, whose tick waits on the gate. + internals.startQueuedMaintenance(); + await waitFor(() => strategy.ticks === 2); + // The maintain joins it, and the pass then meets the failure already reported. + const joining = framework.maintenanceTick(); + strategy.release(); + const joined = await joining; + assert.equal(joined.ran, true); + assert.equal(strategy.ticks, 2, 'the maintain started no pass of its own'); + assert.equal(joined.history.length, 2, 'the pass the maintain joined is kept'); + assert.ok(joined.history[0]!.id > first.history[0]!.id); + } finally { + console.error = quiet; + console.warn = quietWarn; + strategy.release(); + await framework.stop(); + rmSync(dir, { recursive: true, force: true }); + } + }); it('a pass that took a tick before meeting a failure already reported stays in the history', { timeout: 10_000 }, async () => { const dir = mkdtempSync(join(tmpdir(), 'af-maintenance-repeats-after-work-')); const framework = await AgentFramework.create({ @@ -1587,37 +1671,105 @@ describe('queued context maintenance timer', () => { } }); - it("a stop() begun from inside the refusal's alert starts no drain kick", async () => { + it("a stop() begun from inside the refusal's alert waits for the kick the alert reports, which ticks none", { timeout: 10_000 }, async () => { const dir = mkdtempSync(join(tmpdir(), 'af-maintenance-drain-stop-in-alert-')); + const strategy = new GatedTickStrategy(); const framework = await AgentFramework.create({ storePath: join(dir, 'store'), membrane, - agents: [{ name: 'agent', model: 'test-model', systemPrompt: 'test', strategy: new ReadyStrategy() }], + agents: [{ name: 'agent', model: 'test-model', systemPrompt: 'test', strategy }], modules: [], syncIntervalMs: 0, maintenanceIntervalMs: 0, }); + strategy.framework = framework; const internals = framework as unknown as { noteInferenceExhausted(agentName: string, reason: string, retryable?: boolean, errorType?: string): void; overBudgetDrains: Set>; }; + const alerts: string[] = []; + let drainsAtAlert = -1; let stopping: Promise | undefined; // An ops:alert listener that stops the framework, as the alert is raised. - (framework as any).opsAlert = () => { stopping ??= framework.stop(); }; + (framework as any).opsAlert = (kind: string, _agentName: string, message: string) => { + if (kind !== 'context-refusal') return; + alerts.push(message); + drainsAtAlert = internals.overBudgetDrains.size; + stopping ??= framework.stop(); + }; + const logged: string[] = []; const quiet = console.error; - console.error = () => {}; + console.error = (...args: unknown[]) => { logged.push(args.map(String).join(' ')); }; try { internals.noteInferenceExhausted('agent', 'compile refused', false, 'over_budget'); assert.ok(stopping, 'the alert began stop()'); - assert.equal(internals.overBudgetDrains.size, 0, 'no kick was started'); + assert.match(alerts[0] ?? '', /drain breaker kicking;/); + assert.equal(drainsAtAlert, 1, 'the kick joined the drains stop() waits for before the alert'); await stopping; + assert.equal(strategy.ticks, 0, 'the kick ticked none'); + assert.ok( + logged.some((line) => line.includes('drain kicked for agent (OverBudget breaker, 0 ticks, stopped as the framework stops)')), + 'the kick says it stopped, and why', + ); } finally { console.error = quiet; + strategy.release(); await (stopping ?? framework.stop()); rmSync(dir, { recursive: true, force: true }); } }); + it("a surgery begun from inside the refusal's alert waits for the kick the alert reports, which ticks none", { timeout: 10_000 }, async () => { + const dir = mkdtempSync(join(tmpdir(), 'af-maintenance-drain-surgery-in-alert-')); + const strategy = new GatedTickStrategy(); + const framework = await AgentFramework.create({ + storePath: join(dir, 'store'), + membrane, + agents: [{ name: 'agent', model: 'test-model', systemPrompt: 'test', strategy }], + modules: [], + syncIntervalMs: 0, + maintenanceIntervalMs: 0, + }); + strategy.framework = framework; + const cm = framework.getAgent('agent')!.getContextManager(); + const first = String(cm.addMessage('user', [{ type: 'text', text: 'm1' }])); + cm.addMessage('user', [{ type: 'text', text: 'm2' }]); + const internals = framework as unknown as { + noteInferenceExhausted(agentName: string, reason: string, retryable?: boolean, errorType?: string): void; + overBudgetDrains: Set>; + }; + const alerts: string[] = []; + let drainsAtAlert = -1; + let moving: Promise | undefined; + // An ops:alert listener that rolls the agent back, as the alert is raised. + (framework as any).opsAlert = (kind: string, _agentName: string, message: string) => { + if (kind !== 'context-refusal') return; + alerts.push(message); + drainsAtAlert = internals.overBudgetDrains.size; + moving ??= framework.rollbackToMessage('agent', { messageId: first }); + }; + const logged: string[] = []; + const quiet = console.error; + console.error = (...args: unknown[]) => { logged.push(args.map(String).join(' ')); }; + try { + internals.noteInferenceExhausted('agent', 'compile refused', false, 'over_budget'); + assert.ok(moving, 'the alert began a rollback'); + assert.match(alerts[0] ?? '', /drain breaker kicking;/); + assert.equal(drainsAtAlert, 1, 'the kick joined the drains the rollback waits for before the alert'); + await moving; + assert.equal(strategy.ticks, 0, 'the kick ticked none'); + assert.ok( + logged.some((line) => line.includes('drain kicked for agent (OverBudget breaker, 0 ticks, stopped while a surgery holds the store)')), + 'the kick says it stopped, and why', + ); + assert.notEqual(cm.currentBranch().name, 'main', 'the rollback went on once the kick ended'); + } finally { + console.error = quiet; + strategy.release(); + await framework.stop(); + rmSync(dir, { recursive: true, force: true }); + } + }); it("a stop() made from a drain's first tick, before that tick's first await, waits for the tick", { timeout: 10_000 }, async () => { const dir = mkdtempSync(join(tmpdir(), 'af-maintenance-drain-stop-in-tick-')); const strategy = new StopInTickStrategy(); @@ -1640,6 +1792,7 @@ describe('queued context maintenance timer', () => { internals.noteInferenceExhausted('agent', 'compile refused', false, 'over_budget'); await waitFor(() => strategy.stopping !== undefined); await strategy.stopping; + assert.equal(strategy.stoppedBeforeTickEnd, false, 'stop() waited for the tick'); assert.equal(strategy.storeOpenAtTickEnd, true, 'the tick ended before the store closed'); } finally { console.error = quiet; From ae83e883a4b69b4d4c0839d7312f59dcd9e7c4b7 Mon Sep 17 00:00:00 2001 From: Gladys-2413 Date: Fri, 16 Oct 2026 01:14:07 -0400 Subject: [PATCH 10/11] docs(changelog): switchBranch waits for the drain kick too The changelog's new sentence named stop() and an operator surgery as what a drain kick joins before the refusal's alert; switchBranch takes the same hold and waits for the drain the same way, as the paragraph's first sentence and the drain's comment say. Changelog text only. --- changelog.d/cm-0.14.changed.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/changelog.d/cm-0.14.changed.md b/changelog.d/cm-0.14.changed.md index b2ec49603..ff94a701a 100644 --- a/changelog.d/cm-0.14.changed.md +++ b/changelog.d/cm-0.14.changed.md @@ -1,2 +1,2 @@ - A context-maintenance pass no longer fails as a whole when one agent's readiness can't be read. context-manager 0.14, which agent-framework has required since 0.21.1, refuses its readiness queries while a strategy's initialization is pending (inside `switchBranch`, `fork`, `setStrategy` or `ContextManager.open`) or has failed, and a strategy can refuse on its own, as the autobiographical one does for a branch it hasn't reloaded. That agent's run now records the error, the agent gets its own `context-maintenance-failed` alert (`scope: 'agent'`), and the other agents run. Before, such a throw ended the pass for every agent, with one alert for the whole pass (`scope: 'pass'`). Anything else that throws while one agent's maintenance is prepared is that agent's failure in the same way. A failure that lasts, met by every pass, is logged, alerted and recorded in the maintenance history when it's first met, then again when its text changes, every 15 minutes while it lasts (the alert webhook's cooldown for it, counted from the webhook's last post of it), or after that agent's maintenance has gone without a failure. A pass that met only failures already reported, and took no tick, isn't added to the history unless an operator's `maintain` (`maintenanceTick()`) asked for it, so a lasting failure no longer pushes out the runs that did work. `getContextMaintenanceSnapshot()` reports such an agent as `ready: false` with an `error`, and a strategy that can't give a progress snapshot reads as having none. -- While an operator surgery (`rollbackToMessage` or `suppressMessages`) or `switchBranch` holds the store, no maintenance pass starts and no over-budget drain kick is started, a running pass or drain kick starts no new tick, and the operator call waits for them to finish before it switches or forks the branch, so no maintenance work overlaps the move. `stop()` waits for them the same way, a drain kick included, before it closes the store, so the tick in flight ends before the store closes, and once `stop()` has begun, no drain kick starts and none starts a new tick. Maintenance makes no provider call meanwhile, a conversation agent's included: one made under the hold, or once `stop()` has begun, is withdrawn without being sent, and so is a resident's call waiting at its provider gate when the hold is taken or `stop()` begins. That's no maintenance failure, and for a call outside context-manager 0.14's compression fallback ladder, a merge's or a chunk's canonical call with its retries after a `tool_use` stop, context-manager counts no attempt. One inside the ladder, after a chunk's canonical call was refused or cut short, is recorded there as a provider error, and once every rung has failed, the chunk's request family is quarantined, with its alert. Before, a resident's call made once `stop()` had begun was refused as a failure, so a restart that caught a maintenance tick before one of its calls logged a maintenance failure and raised a `context-maintenance-failed` alert; now the agent's maintenance logs that it stopped. An over-budget compile's `context-refusal` alert says whether the drain breaker kicked: `kicking`, `already draining`, or `not kicked` with the reason, where it said `kicking` whenever the agent was found, and `unavailable (no agent handle)` for one it can't find, as before. A kick joins what `stop()` and an operator surgery wait for before the alert is raised, so a listener on the alert that begins either waits for the kick, which then starts no tick. A merge keeps its place in the queue. When a chunk's compression is withdrawn outside the ladder, context-manager logs its "Failed to compress chunk (hierarchical)" line with the withdrawal, and the chunk is queued again at that manager's next rebuild, such as its next message or compile. The operator call never waits on a provider cooldown, or on the retry turn a cooldown queues, which the hold keeps from starting. `switchBranch` can therefore move the store after the call returns, and the API server's `branch.switch` announces the move it made once it has, while `branch.current`, the branch list, `/health` and `/status` read the store's branch whenever they report one. `maintenanceTick()` answers `ran: false` meanwhile, and `resume()` refuses with `OperatorActionError` (`agent-busy`), as the other operator calls do, including when the surgery took the store while `resume()` waited. Resume once the surgery has finished. +- While an operator surgery (`rollbackToMessage` or `suppressMessages`) or `switchBranch` holds the store, no maintenance pass starts and no over-budget drain kick is started, a running pass or drain kick starts no new tick, and the operator call waits for them to finish before it switches or forks the branch, so no maintenance work overlaps the move. `stop()` waits for them the same way, a drain kick included, before it closes the store, so the tick in flight ends before the store closes, and once `stop()` has begun, no drain kick starts and none starts a new tick. Maintenance makes no provider call meanwhile, a conversation agent's included: one made under the hold, or once `stop()` has begun, is withdrawn without being sent, and so is a resident's call waiting at its provider gate when the hold is taken or `stop()` begins. That's no maintenance failure, and for a call outside context-manager 0.14's compression fallback ladder, a merge's or a chunk's canonical call with its retries after a `tool_use` stop, context-manager counts no attempt. One inside the ladder, after a chunk's canonical call was refused or cut short, is recorded there as a provider error, and once every rung has failed, the chunk's request family is quarantined, with its alert. Before, a resident's call made once `stop()` had begun was refused as a failure, so a restart that caught a maintenance tick before one of its calls logged a maintenance failure and raised a `context-maintenance-failed` alert; now the agent's maintenance logs that it stopped. An over-budget compile's `context-refusal` alert says whether the drain breaker kicked: `kicking`, `already draining`, or `not kicked` with the reason, where it said `kicking` whenever the agent was found, and `unavailable (no agent handle)` for one it can't find, as before. A kick joins what `stop()`, an operator surgery and `switchBranch` wait for before the alert is raised, so a listener on the alert that begins any of them waits for the kick, which then starts no tick. A merge keeps its place in the queue. When a chunk's compression is withdrawn outside the ladder, context-manager logs its "Failed to compress chunk (hierarchical)" line with the withdrawal, and the chunk is queued again at that manager's next rebuild, such as its next message or compile. The operator call never waits on a provider cooldown, or on the retry turn a cooldown queues, which the hold keeps from starting. `switchBranch` can therefore move the store after the call returns, and the API server's `branch.switch` announces the move it made once it has, while `branch.current`, the branch list, `/health` and `/status` read the store's branch whenever they report one. `maintenanceTick()` answers `ran: false` meanwhile, and `resume()` refuses with `OperatorActionError` (`agent-busy`), as the other operator calls do, including when the surgery took the store while `resume()` waited. Resume once the surgery has finished. From 192167c40c45f77790dac23ff060a9ab96653292 Mon Sep 17 00:00:00 2001 From: Gladys-2413 Date: Fri, 16 Oct 2026 23:52:55 -0400 Subject: [PATCH 11/11] docs(changelog): a withdrawn chunk is queued again at a sent compile, not a dry one The fragment said a chunk withdrawn outside the fallback ladder is queued again at its manager's next message or compile. A dry-run compile puts context-manager's chunks and compression queue back as the last committing rebuild left them, so it doesn't queue the chunk again, and this repository compiles dry for activation previews and when it creates a conversation agent. The fragment now says "sent compile", as the withdrawal's doc comment already does. Addresses R2-1 of the round-2 review. Co-Authored-By: Claude Opus 5.5 --- changelog.d/cm-0.14.changed.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/changelog.d/cm-0.14.changed.md b/changelog.d/cm-0.14.changed.md index ff94a701a..ea46f82f3 100644 --- a/changelog.d/cm-0.14.changed.md +++ b/changelog.d/cm-0.14.changed.md @@ -1,2 +1,2 @@ - A context-maintenance pass no longer fails as a whole when one agent's readiness can't be read. context-manager 0.14, which agent-framework has required since 0.21.1, refuses its readiness queries while a strategy's initialization is pending (inside `switchBranch`, `fork`, `setStrategy` or `ContextManager.open`) or has failed, and a strategy can refuse on its own, as the autobiographical one does for a branch it hasn't reloaded. That agent's run now records the error, the agent gets its own `context-maintenance-failed` alert (`scope: 'agent'`), and the other agents run. Before, such a throw ended the pass for every agent, with one alert for the whole pass (`scope: 'pass'`). Anything else that throws while one agent's maintenance is prepared is that agent's failure in the same way. A failure that lasts, met by every pass, is logged, alerted and recorded in the maintenance history when it's first met, then again when its text changes, every 15 minutes while it lasts (the alert webhook's cooldown for it, counted from the webhook's last post of it), or after that agent's maintenance has gone without a failure. A pass that met only failures already reported, and took no tick, isn't added to the history unless an operator's `maintain` (`maintenanceTick()`) asked for it, so a lasting failure no longer pushes out the runs that did work. `getContextMaintenanceSnapshot()` reports such an agent as `ready: false` with an `error`, and a strategy that can't give a progress snapshot reads as having none. -- While an operator surgery (`rollbackToMessage` or `suppressMessages`) or `switchBranch` holds the store, no maintenance pass starts and no over-budget drain kick is started, a running pass or drain kick starts no new tick, and the operator call waits for them to finish before it switches or forks the branch, so no maintenance work overlaps the move. `stop()` waits for them the same way, a drain kick included, before it closes the store, so the tick in flight ends before the store closes, and once `stop()` has begun, no drain kick starts and none starts a new tick. Maintenance makes no provider call meanwhile, a conversation agent's included: one made under the hold, or once `stop()` has begun, is withdrawn without being sent, and so is a resident's call waiting at its provider gate when the hold is taken or `stop()` begins. That's no maintenance failure, and for a call outside context-manager 0.14's compression fallback ladder, a merge's or a chunk's canonical call with its retries after a `tool_use` stop, context-manager counts no attempt. One inside the ladder, after a chunk's canonical call was refused or cut short, is recorded there as a provider error, and once every rung has failed, the chunk's request family is quarantined, with its alert. Before, a resident's call made once `stop()` had begun was refused as a failure, so a restart that caught a maintenance tick before one of its calls logged a maintenance failure and raised a `context-maintenance-failed` alert; now the agent's maintenance logs that it stopped. An over-budget compile's `context-refusal` alert says whether the drain breaker kicked: `kicking`, `already draining`, or `not kicked` with the reason, where it said `kicking` whenever the agent was found, and `unavailable (no agent handle)` for one it can't find, as before. A kick joins what `stop()`, an operator surgery and `switchBranch` wait for before the alert is raised, so a listener on the alert that begins any of them waits for the kick, which then starts no tick. A merge keeps its place in the queue. When a chunk's compression is withdrawn outside the ladder, context-manager logs its "Failed to compress chunk (hierarchical)" line with the withdrawal, and the chunk is queued again at that manager's next rebuild, such as its next message or compile. The operator call never waits on a provider cooldown, or on the retry turn a cooldown queues, which the hold keeps from starting. `switchBranch` can therefore move the store after the call returns, and the API server's `branch.switch` announces the move it made once it has, while `branch.current`, the branch list, `/health` and `/status` read the store's branch whenever they report one. `maintenanceTick()` answers `ran: false` meanwhile, and `resume()` refuses with `OperatorActionError` (`agent-busy`), as the other operator calls do, including when the surgery took the store while `resume()` waited. Resume once the surgery has finished. +- While an operator surgery (`rollbackToMessage` or `suppressMessages`) or `switchBranch` holds the store, no maintenance pass starts and no over-budget drain kick is started, a running pass or drain kick starts no new tick, and the operator call waits for them to finish before it switches or forks the branch, so no maintenance work overlaps the move. `stop()` waits for them the same way, a drain kick included, before it closes the store, so the tick in flight ends before the store closes, and once `stop()` has begun, no drain kick starts and none starts a new tick. Maintenance makes no provider call meanwhile, a conversation agent's included: one made under the hold, or once `stop()` has begun, is withdrawn without being sent, and so is a resident's call waiting at its provider gate when the hold is taken or `stop()` begins. That's no maintenance failure, and for a call outside context-manager 0.14's compression fallback ladder, a merge's or a chunk's canonical call with its retries after a `tool_use` stop, context-manager counts no attempt. One inside the ladder, after a chunk's canonical call was refused or cut short, is recorded there as a provider error, and once every rung has failed, the chunk's request family is quarantined, with its alert. Before, a resident's call made once `stop()` had begun was refused as a failure, so a restart that caught a maintenance tick before one of its calls logged a maintenance failure and raised a `context-maintenance-failed` alert; now the agent's maintenance logs that it stopped. An over-budget compile's `context-refusal` alert says whether the drain breaker kicked: `kicking`, `already draining`, or `not kicked` with the reason, where it said `kicking` whenever the agent was found, and `unavailable (no agent handle)` for one it can't find, as before. A kick joins what `stop()`, an operator surgery and `switchBranch` wait for before the alert is raised, so a listener on the alert that begins any of them waits for the kick, which then starts no tick. A merge keeps its place in the queue. When a chunk's compression is withdrawn outside the ladder, context-manager logs its "Failed to compress chunk (hierarchical)" line with the withdrawal, and the chunk is queued again at that manager's next rebuild, such as its next message or sent compile. The operator call never waits on a provider cooldown, or on the retry turn a cooldown queues, which the hold keeps from starting. `switchBranch` can therefore move the store after the call returns, and the API server's `branch.switch` announces the move it made once it has, while `branch.current`, the branch list, `/health` and `/status` read the store's branch whenever they report one. `maintenanceTick()` answers `ran: false` meanwhile, and `resume()` refuses with `OperatorActionError` (`agent-busy`), as the other operator calls do, including when the surgery took the store while `resume()` waited. Resume once the surgery has finished.