fix(code-execution): separate script execution from bounded observation

changelog.d/code-execution-lifecycle.added.md

@@ -0,0 +1,3 @@
+- Retain execution IDs/results for bounded waits, listing and cancellation. Expose
+  a framework method for the host's admin-only release command to unblock an
+  observation without killing its script, then wake on completion.

changelog.d/code-execution-lifecycle.breaking.md

@@ -0,0 +1,5 @@
+- **Callers of `code_execution`:** foreground calls now return a running `script_id`
+  after a bounded observation budget (10 seconds by default). Check `status` and
+  use `action=wait` for the result; timeout no longer monopolizes inference until
+  script termination. `on_timeout=end_turn` arms completion notification and
+  releases the turn. Explicit cancellation remains separate.

changelog.d/code-execution-lifecycle.fixed.md

@@ -0,0 +1,5 @@
+- Prevent concurrent interpreter startup from losing an execution; settle startup
+  cancellation and reject stale tool/wake replies after interpreter replacement.
+  Preserve sub-second Python tool timeouts.
+- Serialize background wake delivery across rate limits and caps; report delivery
+  failure to Python. Keep deferred end-turn effects scoped to their execution.

docs/code-execution-lifecycle.md

@@ -0,0 +1,121 @@
+# Code execution: execution, observation, and attention
+
+A running operation and a waiting agent have different lifetimes. Python's
+`await` suspends the coroutine; it should not oblige the agent to spend its
+whole turn waiting. Keep Python semantics intact and put a bounded observation
+around execution in the harness.
+
+This branch implements that separation while preserving existing Python
+contexts and background watchers. It is an initial lifecycle change, not a
+durable job service or a replacement for the terminal execution protocol.
+
+## Agent interface implemented here
+
+One tool, `code_execution`, retains `code`, `background`, `action`, and
+`script_id`, and adds `wait_ms` and `on_timeout`:
+
+```json
+{"code":"import asyncio\nawait asyncio.sleep(30)\nprint('done')","wait_ms":1000}
+```
+
+Quick results return stdout, stderr, and return_code as before, with an
+execution ID and status. If still running after the observation budget, the
+tool returns that ID with status `running`. The script continues, and its
+completion notifies the owner. `action=wait` retrieves the retained result or
+observes the same operation for another bounded interval. `wait_ms=0` returns
+immediately; the maximum is 60 seconds. The default is 10 seconds, configurable
+through agent-framework's `codeExecution.foregroundWaitMs`.
+
+```json
+{"action":"wait","script_id":"py-1","wait_ms":1000,"on_timeout":"end_turn"}
+```
+
+If the result arrives within the budget, the agent receives it and continues.
+Otherwise, completion notification is armed **before** the tool result requests
+`endTurn`. This uses the scheduler's existing result boundary: the stream ends
+without cancelling the script. Completion is queued even if it arrives during
+turn teardown. It bypasses ambient event gating because the agent armed it.
+This ends a turn, not a timed gate sleep: other authorized events can still wake
+the agent before the script finishes.
+
+Repeated waits refer to one execution and never replay its side effects. A
+wait already present at completion receives the result directly, suppressing
+the additional completion wake. Multiple expired waits arm one completion
+notice. The notice contains a bounded tail; the full captured result can be
+retrieved with `wait` and uses the normal spill policy. Ordinary foreground
+stdout is currently available at completion, not streamed during execution.
+
+The existing foreground interpreter remains persistent and serial. A second
+run while it is busy fails with the running ID and recovery instructions.
+`background=true` uses an independent interpreter, retains `wake_agent`, and
+journals output to the workspace. Its default remains immediate return and
+silent clean exit; specifying a wait budget uses the same observation policy.
+Background crashes still notify. `list` reports both modes; the legacy
+`background_scripts` field is preserved. `cancel` stops Python explicitly.
+
+Ephemeral agents cannot request `on_timeout=end_turn`: their owner is destroyed
+when the turn ends and cannot receive the promised wake. Owner disposal cancels
+its executions and releases their interpreters. Background watchers retain the
+existing primary-agent restriction.
+
+## Operator interface implemented here
+
+The connectome-host companion adds the admin command `/release-wait [script_id]`
+alongside `/undo`. With no ID it releases all current code-execution observations
+for the selected agent. With an ID it releases only that execution's observers.
+It does not abort Python or reset the agent. The underlying framework method is
+`releaseCodeExecutionWait(agentName, scriptId?)`.
+
+This is an admin operation, not an agent tool. The host requires trusted command
+provenance: the local operator CLI/TUI or a full-authority web client. Generic
+headless/fleet IPC and read-only web observers cannot invoke it. A caller cannot
+grant itself admin status with an argument in the slash command. The framework's
+generic socket API does not expose a release command.
+
+A release after completion or when no observation is active returns `released: 0`;
+it does not end an unrelated turn. Releasing a code observation does not settle
+unrelated tools in the same batch. This is not a general-purpose inference reset.
+
+## Correctness changes
+
+- Reserve the Python runner before interpreter startup, so simultaneous cold
+  calls cannot overwrite one another and startup can be cancelled.
+- Bind asynchronous tool replies and wake acknowledgements to the originating
+  interpreter and execution. A late reply must not satisfy a replacement
+  interpreter's reused `t1` or `w1` ID.
+- Preserve fractional seconds when sending inner-tool timeouts to Python.
+- Serialize each script's wake requests, enforcing the interval and cap under
+  `asyncio.gather`; stop rate-limit waits when execution ends.
+- Acknowledge `wake_agent` only after context delivery and inference enqueue
+  succeed. Delivery failures are errors visible to Python.
+- Scope inner-tool end-turn requests to their execution, preventing background
+  or late results from ending an unrelated foreground call.
+
+## Limits and the next design steps
+
+Execution IDs and the five most recently settled results per owner are held in
+memory. Host restart loses them and stops Python. Completion delivery is not a
+durable receipt/outbox: storage failure is logged, and an operator can retrieve
+the result while the host remains alive. Explicit script wakes report delivery
+failure to Python. Cancellation is not rollback, and does not cancel an already
+dispatched inner tool. There is no claim of exactly-once external effects.
+
+The next increment should give operations a bounded, cursor-addressed journal
+and persistent terminal outcomes. Associate explicit notifications with delivery
+receipts: queued, delivered at a tool boundary, or used to start a fresh turn.
+That would avoid an unnecessary subsequent turn when an active inference has
+already consumed the completion event. The existing wake scheduler can currently
+queue such a follow-up. Crash recovery should report an interrupted operation,
+not automatically rerun side-effectful code.
+
+Keep routine output in the journal; only explicit signals, failures, and
+requested completions should request attention. Waiting on several operations
+should eventually offer “any” and “all” without polling code. Add named Python
+contexts only when workflows need several persistent contexts; arbitrary Python
+globals and stdout cannot safely be shared by concurrent top-level scripts.
+
+For MCPL, this should be an operation lifecycle with ownership and cancellation
+capabilities, separate from inference lifecycle metadata. A terminal operation
+should retain its own run ID and completion even if Python or the model turn
+ends. The harness observes that operation; a server should not need to guess
+whether a timeout meant “stop waiting,” “stop executing,” or “go idle.”

src/code-execution/py-runner.ts

@@ -139,36 +139,21 @@ export class PyRunner {
       };
     }
     this.clearIdleTimer();
-
-    try {
-      await this.ensureChild();
-    } catch (err) {
-      const message = err instanceof Error ? err.message : String(err);
-      this.reclaim('spawn-failed');
-      return {
-        stdout: '',
-        stderr: `Failed to start python runtime (${this.pythonPath}): ${message}`,
-        returnCode: 1,
-        aborted: true,
-      };
-    }
-
     const execId = `e${++this.execCounter}`;
     const deadlineMs = background?.lifetimeMs ?? this.scriptTimeoutMs;
     this.onWake = background?.onWake ?? null;
-    const result = await new Promise<ExecResult>((resolve) => {
-      const pending: PendingExec = {
-        id: execId,
-        resolve,
-        deadlineTimer: null,
-        killTimer: null,
-        settled: false,
-      };
-      this.pending = pending;
-
+    let resolve!: (result: ExecResult) => void;
+    const completion = new Promise<ExecResult>((r) => { resolve = r; });
+    // Reserve before the first await. Startup is part of the execution:
+    // concurrent calls must not overwrite its result resolver, and abort /
+    // dispose must be able to settle it even before Python says ready.
+    const pending: PendingExec = {
+      id: execId, resolve, deadlineTimer: null, killTimer: null, settled: false,
+    };
+    this.pending = pending;
+    void this.ensureChild().then(() => {
+      if (this.pending !== pending || pending.settled) return;
       pending.deadlineTimer = setTimeout(() => {
-        // Deadline: ask politely first (script sees CancelledError and its
-        // exec_result still flows back), then kill on unresponsiveness.
         this.send({ op: 'cancel', id: execId, reason: 'deadline' });
         pending.killTimer = setTimeout(() => {
           this.settlePending({
@@ -180,22 +165,31 @@ export class PyRunner {
           this.reclaim('deadline-kill');
         }, CANCEL_GRACE_MS);
       }, deadlineMs);
-      // A day-scale background deadline must not hold the process open.
       if (background) pending.deadlineTimer.unref?.();
-
       this.send({
         op: 'init',
         tools: tools.map((t) => ({ py_name: t.pyName, tool_name: t.toolName })),
-        call_timeout_s: Math.round(this.toolCallTimeoutMs / 1000),
-        ...(background
-          ? { background: true, log_path: background.logPath ?? null }
-          : {}),
+        call_timeout_s: this.toolCallTimeoutMs / 1000,
+        ...(background ? { background: true, log_path: background.logPath ?? null } : {}),
       });
       this.send({ op: 'exec', id: execId, code });
+    }).catch((err) => {
+      if (this.pending !== pending || pending.settled) return;
+      const message = err instanceof Error ? err.message : String(err);
+      this.settlePending({
+        stdout: '',
+        stderr: `Failed to start python runtime (${this.pythonPath}): ${message}`,
+        returnCode: 1,
+        aborted: true,
+      });
+      this.reclaim('spawn-failed');
     });
+    const result = await completion;
 
-    this.onWake = null;
-    if (!background) this.armIdleTimer();
+    if (!this.pending) {
+      this.onWake = null;
+      if (!background) this.armIdleTimer();
+    }
     return result;
   }
 
@@ -261,6 +255,7 @@ export class PyRunner {
     });
 
     child.on('exit', (exitCode, signal) => {
+      if (this.child !== child) return;
       if (this.pending) {
         this.settlePending({
           stdout: '',
@@ -275,7 +270,9 @@ export class PyRunner {
     });
 
     this.reader = createInterface({ input: child.stdout });
-    this.reader.on('line', (line) => this.handleLine(line));
+    this.reader.on('line', (line) => {
+      if (this.child === child) this.handleLine(line);
+    });
 
     this.childReady = new Promise<void>((resolve, reject) => {
       const onReady = () => {
@@ -297,20 +294,23 @@ export class PyRunner {
       const cleanup = () => {
         clearTimeout(timeout);
         this.readyResolver = null;
+        this.readyRejecter = null;
         child.off('exit', onExit);
         child.off('error', onError);
       };
       this.readyResolver = onReady;
+      this.readyRejecter = onError;
       child.on('exit', onExit);
       child.on('error', onError);
     });
     return this.childReady;
   }
 
   private readyResolver: (() => void) | null = null;
+  private readyRejecter: ((error: Error) => void) | null = null;
 
   private handleLine(line: string): void {
-    let msg: { op?: string; id?: string; name?: string; args?: unknown; stdout?: string; stderr?: string; return_code?: number };
+    let msg: { op?: string; id?: string; exec_id?: string; name?: string; args?: unknown; stdout?: string; stderr?: string; return_code?: number };
     try {
       msg = JSON.parse(line);
     } catch {
@@ -324,6 +324,9 @@ export class PyRunner {
         return;
 
       case 'tool_call': {
+        const child = this.child;
+        const pending = this.pending;
+        if (!pending || msg.exec_id !== pending.id) return;
         const callId = msg.id;
         const toolName = msg.name;
         if (!callId || !toolName) return;
@@ -334,12 +337,19 @@ export class PyRunner {
         this.onToolCall(toolName, args)
           .catch((err) => `Error: ${err instanceof Error ? err.message : String(err)}`)
           .then((result) => {
-            this.send({ op: 'tool_result', id: callId, result });
+            // A reclaimed interpreter starts call ids again at t1. Late
+            // results from its predecessor must never satisfy the new call.
+            if (this.child === child && this.pending === pending) {
+              this.send({ op: 'tool_result', id: callId, result });
+            }
           });
         return;
       }
 
       case 'wake': {
+        const child = this.child;
+        const pending = this.pending;
+        if (!pending || msg.exec_id !== pending.id) return;
         const wakeId = msg.id;
         if (!wakeId) return;
         const line = typeof (msg as { line?: unknown }).line === 'number'
@@ -352,7 +362,9 @@ export class PyRunner {
               `wake handler failed: ${err instanceof Error ? err.message : String(err)}`)
           : Promise.resolve('this script is not allowed to wake the agent');
         void refuse.then((error) => {
-          this.send({ op: 'wake_ack', id: wakeId, ...(error ? { error } : {}) });
+          if (this.child === child && this.pending === pending) {
+            this.send({ op: 'wake_ack', id: wakeId, ...(error ? { error } : {}) });
+          }
         });
         return;
       }
@@ -408,6 +420,7 @@ export class PyRunner {
   }
 
   private teardownChild(): void {
+    this.readyRejecter?.(new Error('python runtime reclaimed during startup'));
     if (this.reader) {
       this.reader.close();
       this.reader = null;

src/code-execution/runtime-py.ts

@@ -196,7 +196,7 @@ def _make_tool_fn(tool_name, py_name):
         except asyncio.TimeoutError:
             raise TimeoutError(
                 "Calling tool ['" + tool_name + "'] timed out (no response after "
-                + str(int(CALL_TIMEOUT_S)) + "s)."
+                + str(CALL_TIMEOUT_S) + "s)."
             )
         finally:
             _pending_tool_futures.pop(call_id, None)

src/code-execution/script-run.ts

@@ -0,0 +1,54 @@
+import type { ExecResult } from './py-runner.js';
+
+export type TimeoutPolicy = 'continue' | 'end_turn';
+export interface ScriptObservation {
+  result?: ExecResult;
+  endTurn: boolean;
+}
+
+/** A script's lifetime is independent of any one caller's observation budget.
+ * One completion notice is armed when an observation times out or is released.
+ * An observer present at completion receives the result directly instead.
+ */
+export class ScriptRun {
+  result: ExecResult | undefined;
+  private waiters = new Set<(endTurn?: boolean) => void>();
+  private notifyOnCompletion = false;
+
+  constructor(
+    completion: Promise<ExecResult>,
+    onComplete: (result: ExecResult, notify: boolean, observed: boolean) => void,
+  ) {
+    void completion.then((result) => {
+      this.result = result;
+      const observed = this.waiters.size > 0;
+      for (const finish of [...this.waiters]) finish();
+      onComplete(result, this.notifyOnCompletion && !observed, observed);
+    });
+  }
+
+  get observing(): boolean { return this.waiters.size > 0; }
+
+  observe(waitMs: number, onTimeout: TimeoutPolicy): Promise<ScriptObservation> {
+    if (this.result) return Promise.resolve({ result: this.result, endTurn: false });
+    return new Promise((resolve) => {
+      let timer: ReturnType<typeof setTimeout> | undefined;
+      const finish = (endTurn = false) => {
+        if (!this.waiters.delete(finish)) return;
+        clearTimeout(timer);
+        if (!this.result) this.notifyOnCompletion = true;
+        resolve({ result: this.result, endTurn: !this.result && endTurn });
+      };
+      this.waiters.add(finish);
+      if (waitMs === 0) finish(onTimeout === 'end_turn');
+      else timer = setTimeout(() => finish(onTimeout === 'end_turn'), waitMs);
+    });
+  }
+
+  /** Operator rescue: end only the observation/turn, never the script. */
+  release(): number {
+    const count = this.waiters.size;
+    for (const finish of [...this.waiters]) finish(true);
+    return count;
+  }
+}