-
Notifications
You must be signed in to change notification settings - Fork 15
fix: user interrupt is a cancellation, not a failure (+ honest no-locus marker, loud gate-buffer drop) #134
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,15 @@ | ||
| - A user-initiated stream stop (host Stop button / `cancelStream()`) is no | ||
| longer recorded as an inference failure. It now emits `inference:aborted` | ||
| (reason `user`) instead of `inference:exhausted`, leaves the | ||
| consecutive-failure streak and ops alerts untouched, and writes an honest | ||
| `[turn-interrupted]` chronicle marker ("deliberate cancellation, not a | ||
| failure") in place of the misleading `[inference-failed] the model call | ||
| failed…` text with remediation advice for a failure that never happened. | ||
| - Speech-route failures with no delivery locus (headless/WebUI turns with no | ||
| home or trigger channel) now read `[send-undeliverable] … had no delivery | ||
| destination` instead of claiming a Discord delivery failure to "the | ||
| channel". The machine-readable marker `kind` is unchanged. | ||
| - The event gate's inference buffer no longer drops its oldest pending event | ||
| silently when full: the drop is logged to stderr with the policy and event | ||
| type. A dropped event never triggers inference, so a silent drop looked | ||
| like "message sent, never answered, queue depth 0" from the outside. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -6872,6 +6872,17 @@ export class AgentFramework { | |
| } | ||
| } | ||
| const reason = event.reason ?? 'unknown'; | ||
| // Membrane emits reason 'user' exactly when someone called | ||
| // stream.cancel() — the host's Stop button / Escape / an admin | ||
| // abort. That is a DELIBERATE cancellation, not a failure: it | ||
| // must not feed the consecutive-failure streak, ops alerts, or | ||
| // the "[inference-failed] the model call failed" chronicle | ||
| // marker, all of which told the agent (in the user's voice!) | ||
| // that its turn failed and advised remediation for a failure | ||
| // that never happened. For long-lived residents whose transcript | ||
| // is memory, those mislabeled cancellations accumulate as false | ||
| // self-knowledge. | ||
| const deliberate = reason === 'user'; | ||
| // Only reset if this is still the active stream (a budget restart | ||
| // may have already started a new stream, bumping streamId) | ||
| if (agent.streamId === myStreamId) { | ||
|
|
@@ -6881,13 +6892,45 @@ export class AgentFramework { | |
| this.settleAgent(agent.name, { | ||
| stopReason: 'exhausted', | ||
| speech: '', | ||
| error: `Stream aborted: ${reason}`, | ||
| }); | ||
| this.emitTrace({ | ||
| type: 'inference:exhausted', | ||
| agentName: agent.name, | ||
| error: `Stream aborted: ${reason}`, | ||
| error: deliberate ? 'Stream stopped by user' : `Stream aborted: ${reason}`, | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Prompt To Fix With AIThis is a comment left during a code review.
Path: src/framework.ts
Line: 6895
Comment:
**Ephemeral cancellation still rejects** If a user stops an ephemeral agent’s stream, this branch still settles it as `exhausted`. That makes `settleAgent` reject the run promise, so a caller awaiting `runEphemeralToCompletion` receives an error for a deliberate cancellation despite the new aborted trace.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly. |
||
| }); | ||
| if (deliberate) { | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Prompt To Fix With AIThis is a comment left during a code review.
Path: src/framework.ts
Line: 6897
Comment:
**Stops remain health errors** Although this branch avoids the failure trace, it still reaches the inference-log write with `success: false` and `Stream aborted: user`. A deliberate stop consequently appears in `errorsOnly` queries and in health snapshots’ error counts and recent errors, leaving the failure classification in those durable records.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly. |
||
| // Distinct trace type: emitTrace funnels every | ||
| // inference:exhausted into noteInferenceExhausted (streak, | ||
| // failures.log, marker); inference:aborted carries the | ||
| // honest cause without any of that. | ||
| this.emitTrace({ | ||
| type: 'inference:aborted', | ||
|
Comment on lines
+6897
to
+6903
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Prompt To Fix With AIThis is a comment left during a code review.
Path: src/framework.ts
Line: 6897-6903
Comment:
**Cancelled inference reports completion** On a user stop, this branch emits an aborted trace but leaves the MCPL lifecycle phase at its default of `completed`. When the stream exits, subscribed servers are told the cancelled inference completed.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly. |
||
| agentName: agent.name, | ||
| reason: 'user', | ||
| durationMs, | ||
|
Comment on lines
+6902
to
+6906
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Prompt To Fix With AIThis is a comment left during a code review.
Path: src/framework.ts
Line: 6902-6906
Comment:
**Abort emits duplicate traces** For a streaming agent, `framework.abortInference()` already emits `inference:aborted` with the caller’s reason before cancelling the stream. This handler emits it again when the stream reports its abort, this time with reason `user`. Subscribers receive two terminal traces for one stop, potentially with conflicting reasons.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly. |
||
| }); | ||
| // Agent-facing marker, honest about what happened and who | ||
| // acted (no inference triggered → no loop). Same system | ||
| // envelope as the failure marker so surfaces render it | ||
| // the same way. | ||
| try { | ||
| agent.getContextManager().addMessage( | ||
| 'user', | ||
| [{ | ||
| type: 'text', | ||
| text: | ||
| `[turn-interrupted] Your previous turn was stopped mid-stream ` + | ||
| `by the user — a deliberate cancellation, not a failure. Any ` + | ||
| `partial output was cut off by the stop and was not delivered.`, | ||
|
Comment on lines
+6918
to
+6920
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Prompt To Fix With AIThis is a comment left during a code review.
Path: src/framework.ts
Line: 6918-6920
Comment:
**Delivered output called undelivered** If an earlier tool round has already sent prose to a channel when the user presses Stop, that delivery is not rolled back. The new marker nevertheless says partial output was not delivered, which can lead the agent to repeat something the human already received.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly. |
||
| }], | ||
| { system: true, kind: 'turn-interrupted', reason }, | ||
| ); | ||
| } catch (err) { | ||
| console.error(`[turn-interrupted] could not record chronicle marker for ${agent.name}:`, err); | ||
| } | ||
| } else { | ||
| this.emitTrace({ | ||
| type: 'inference:exhausted', | ||
| agentName: agent.name, | ||
| error: `Stream aborted: ${reason}`, | ||
| }); | ||
| } | ||
| // Postmortem 2026-05-28 P2 #7: persist the abort to the | ||
| // inference log so future investigations can attribute the | ||
| // terminal cause without relying on live in-memory reducer | ||
|
|
@@ -8904,12 +8947,19 @@ export class AgentFramework { | |
| ? `${label.startsWith('#') ? label : `#${label}`} (${channelId})` | ||
| : channelId | ||
| : 'the channel'; | ||
| // No-locus failures (headless/WebUI turns with no home or | ||
| // trigger channel) are not Discord failures: a "[discord-send- | ||
| // failed] could not be delivered to the channel" marker sent | ||
| // the agent debugging a Discord problem that doesn't exist. | ||
| // The machine-readable `kind` stays stable — the gate's | ||
| // discord-send-failed-skip intent keys on it — but the text | ||
| // the agent reads names the real situation. | ||
| const text = channelId | ||
| ? `[discord-send-failed] Your previous reply (${textLen} chars) could not be delivered to ${where} (${reason}). It was saved to your archive but the human did not receive it.` | ||
| : `[send-undeliverable] Your previous reply (${textLen} chars) had no delivery destination — ${reason}. This is a routing/configuration situation, not a channel failure. The reply was saved to your archive but was not delivered anywhere.`; | ||
| this.addMessage( | ||
| 'user', | ||
| [{ | ||
| type: 'text', | ||
| text: `[discord-send-failed] Your previous reply (${textLen} chars) could not be delivered to ${where} (${reason}). It was saved to your archive but the human did not receive it.`, | ||
| }], | ||
| [{ type: 'text', text }], | ||
| { system: true, kind: 'discord-send-failed', channelId: channelId ?? '', reason }, | ||
| ); | ||
| } catch (err) { | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,186 @@ | ||
| import { describe, it } from 'node:test'; | ||
| import assert from 'node:assert/strict'; | ||
| import { mkdtempSync, rmSync } from 'node:fs'; | ||
| import { tmpdir } from 'node:os'; | ||
| import { join } from 'node:path'; | ||
| import type { | ||
| EventResponse, | ||
| Module, | ||
| ModuleContext, | ||
| ProcessEvent, | ||
| ProcessState, | ||
| ToolCall, | ||
| ToolDefinition, | ||
| ToolResult, | ||
| TraceEvent, | ||
| } from '../src/index.js'; | ||
| import { AgentFramework } from '../src/index.js'; | ||
| import { createMockResponse, MockMembrane } from './helpers/mock-membrane.js'; | ||
|
|
||
| /** | ||
| * A user pressing Stop is a deliberate cancellation, not a model failure. | ||
| * Before the fix, the host's cancelStream() path fell through driveStream's | ||
| * generic abort handling into the failure pipeline: an inference:exhausted | ||
| * trace, a bumped consecutive-failure streak (three Stops → hard-down ops | ||
| * alert), and an "[inference-failed] the model call failed and produced no | ||
| * response … drop an oversized attachment" chronicle marker attributed to | ||
| * the user — three inaccuracies (wrong cause, wrong speaker, irrelevant | ||
| * advice) accumulating as false self-knowledge in a resident's transcript. | ||
| */ | ||
|
|
||
| /** Module whose tool call hangs until released — keeps the stream open so | ||
| * the test can cancel mid-turn, exactly as the TUI/WebUI Stop button does. */ | ||
| class HangingToolModule implements Module { | ||
| readonly name = 'test'; | ||
| release!: () => void; | ||
| private readonly gate = new Promise<void>((resolve) => { this.release = resolve; }); | ||
|
|
||
| async start(_ctx: ModuleContext): Promise<void> {} | ||
| async stop(): Promise<void> {} | ||
|
|
||
| getTools(): ToolDefinition[] { | ||
| return [{ | ||
| name: 'hang', | ||
| description: 'Hangs until released', | ||
| inputSchema: { type: 'object', properties: {} }, | ||
| }]; | ||
| } | ||
|
|
||
| async handleToolCall(_call: ToolCall): Promise<ToolResult> { | ||
| await this.gate; | ||
| return { success: true, data: {} }; | ||
| } | ||
|
|
||
| async onProcess(event: ProcessEvent, _state: ProcessState): Promise<EventResponse> { | ||
| if (event.type === 'external-message') { | ||
| return { | ||
| addMessages: [{ participant: 'User', content: [{ type: 'text', text: String(event.content) }] }], | ||
| requestInference: true, | ||
| }; | ||
| } | ||
| return {}; | ||
| } | ||
| } | ||
|
|
||
| async function waitFor(cond: () => boolean, ms = 2000): Promise<void> { | ||
| const start = Date.now(); | ||
| while (!cond()) { | ||
| if (Date.now() - start > ms) throw new Error('timeout waiting for condition'); | ||
| await new Promise((r) => setTimeout(r, 10)); | ||
| } | ||
| } | ||
|
|
||
| describe('user interrupt is not recorded as a failure', () => { | ||
| it('cancelStream mid-turn → [turn-interrupted] marker, inference:aborted trace, no failure streak', async () => { | ||
| const tempDir = mkdtempSync(join(tmpdir(), 'interrupt-')); | ||
| const membrane = new MockMembrane(); | ||
| // One response with a hanging tool call: the stream stays alive in | ||
| // waiting_for_tools until we cancel it. | ||
| membrane.pushResponse(createMockResponse( | ||
| [{ type: 'tool_use', id: 't1', name: 'test--hang', input: {} } as never], | ||
| 'tool_use', | ||
| )); | ||
|
|
||
| const module = new HangingToolModule(); | ||
| const framework = await AgentFramework.create({ | ||
| storePath: join(tempDir, 'test.chronicle'), | ||
| membrane: membrane.asMembrane(), | ||
| agents: [{ name: 'assistant', model: 'test-model', systemPrompt: 'Assist.' }], | ||
| modules: [module], | ||
| }); | ||
|
|
||
| const traces: TraceEvent[] = []; | ||
| framework.onTrace((t) => { traces.push(t); }); | ||
|
|
||
| try { | ||
| framework.pushEvent({ type: 'external-message', source: 'test', content: 'go', metadata: {} }); | ||
| framework.start(); | ||
|
|
||
| const agent = framework.getAgent('assistant')!; | ||
| await waitFor(() => agent.state.status === 'waiting_for_tools'); | ||
|
|
||
| // What the TUI / WebUI Stop button does. | ||
| agent.cancelStream(); | ||
|
|
||
| await waitFor(() => traces.some((t) => t.type === 'inference:aborted')); | ||
| // Let the abort settle fully (chronicle marker write). | ||
| await waitFor(() => { | ||
| const { messages } = agent.getContextManager().queryMessages({}); | ||
| return messages.some((m) => | ||
| m.content.some((b) => b.type === 'text' && b.text.includes('[turn-interrupted]'))); | ||
| }); | ||
|
|
||
| const { messages } = agent.getContextManager().queryMessages({}); | ||
| const texts = messages.flatMap((m) => | ||
| m.content.filter((b): b is { type: 'text'; text: string } => b.type === 'text').map((b) => b.text)); | ||
|
|
||
| // Honest marker present… | ||
| const marker = texts.find((t) => t.includes('[turn-interrupted]')); | ||
| assert.ok(marker, 'expected a [turn-interrupted] chronicle marker'); | ||
| assert.match(marker!, /deliberate cancellation, not a failure/); | ||
| // …and no failure framing anywhere. | ||
| assert.ok(!texts.some((t) => t.includes('[inference-failed]')), | ||
| 'a user stop must not produce an [inference-failed] marker'); | ||
|
|
||
| // Trace: aborted (with the honest reason), not exhausted. | ||
| const aborted = traces.find((t) => t.type === 'inference:aborted') as { reason?: string }; | ||
| assert.equal(aborted?.reason, 'user'); | ||
| assert.ok(!traces.some((t) => t.type === 'inference:exhausted'), | ||
| 'a user stop must not emit inference:exhausted (feeds streak + ops alerts)'); | ||
| } finally { | ||
| // Release the hung tool BEFORE stopping: its completion pushes a | ||
| // tool-result event, which must land while the queue is still open. | ||
| module.release(); | ||
| await new Promise((r) => setTimeout(r, 50)); | ||
| await framework.stop(); | ||
| rmSync(tempDir, { recursive: true, force: true }); | ||
| } | ||
| }); | ||
|
|
||
| it('a real provider abort (non-user reason) still goes through the failure pipeline', async () => { | ||
| const tempDir = mkdtempSync(join(tmpdir(), 'interrupt-real-')); | ||
| const membrane = new MockMembrane(); | ||
| membrane.pushResponse(createMockResponse( | ||
| [{ type: 'tool_use', id: 't1', name: 'test--hang', input: {} } as never], | ||
| 'tool_use', | ||
| )); | ||
|
|
||
| const module = new HangingToolModule(); | ||
| const framework = await AgentFramework.create({ | ||
| storePath: join(tempDir, 'test.chronicle'), | ||
| membrane: membrane.asMembrane(), | ||
| agents: [{ name: 'assistant', model: 'test-model', systemPrompt: 'Assist.' }], | ||
| modules: [module], | ||
| }); | ||
|
|
||
| const traces: TraceEvent[] = []; | ||
| framework.onTrace((t) => { traces.push(t); }); | ||
|
|
||
| try { | ||
| framework.pushEvent({ type: 'external-message', source: 'test', content: 'go', metadata: {} }); | ||
| framework.start(); | ||
|
|
||
| const agent = framework.getAgent('assistant')!; | ||
| await waitFor(() => agent.state.status === 'waiting_for_tools'); | ||
|
|
||
| // Simulate a provider-side abort: emit the event with a non-user | ||
| // reason directly on the live mock stream. | ||
| const stream = membrane.lastStream!; | ||
| (stream as unknown as { events: unknown[]; pendingResolve: (() => void) | null }).events.push( | ||
| { type: 'aborted', reason: 'connection_lost' }); | ||
| const pr = (stream as unknown as { pendingResolve: (() => void) | null }).pendingResolve; | ||
| if (pr) { (stream as unknown as { pendingResolve: null }).pendingResolve = null; pr(); } | ||
|
|
||
| await waitFor(() => traces.some((t) => t.type === 'inference:exhausted')); | ||
| const exhausted = traces.find((t) => t.type === 'inference:exhausted') as { error?: string }; | ||
| assert.match(exhausted?.error ?? '', /connection_lost/); | ||
| } finally { | ||
| // Release the hung tool BEFORE stopping: its completion pushes a | ||
| // tool-result event, which must land while the queue is still open. | ||
| module.release(); | ||
| await new Promise((r) => setTimeout(r, 50)); | ||
| await framework.stop(); | ||
| rmSync(tempDir, { recursive: true, force: true }); | ||
| } | ||
| }); | ||
| }); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
framework.stop()cancels an active stream, it does not register a framework-cancellation marker. This branch therefore records that the user stopped the turn, even though the host was shutting down. The agent is left with a false account of who acted.Prompt To Fix With AI