diff --git a/changelog.d/fix-code-execution-output-to-host.breaking.md b/changelog.d/fix-code-execution-output-to-host.breaking.md new file mode 100644 index 000000000..9939a5827 --- /dev/null +++ b/changelog.d/fix-code-execution-output-to-host.breaking.md @@ -0,0 +1 @@ +- **Hosts:** the `code_execution` interpreter now takes its channel to the host on its file descriptors 3 and 4, and `PYTHON_RUNTIME_SOURCE`, which the package exports, speaks a new protocol there. **Who needs to act:** a host whose `pythonPath` starts Python through a wrapper that doesn't pass fds 3 and 4 on, such as one that runs it in a container, under sudo with its default settings, or from a Python script through `subprocess`, whose `close_fds` is on by default, and a host that runs `PYTHON_RUNTIME_SOURCE` itself rather than through `PyRunner`. Until it acts, every script fails to start, and the result says the host's channel didn't reach the interpreter: the runtime says so on stderr and exits with 78 (EX_CONFIG). **Migration:** have the wrapper `exec` Python, or pass fds 3 and 4 on. A host that runs the runtime itself opens fd 3 for its messages to the runtime and fd 4 for the runtime's, gives it `/dev/null` as fd 0, reads its fds 1 and 2 itself, puts a `mark` in each `exec`, and takes the script's output on each stream up to that mark, which the runtime writes once the script returns: `exec_result` now carries only the return code. It stops the runtime with the `exit` op, since closing fd 3 while a script runs now ends the runtime with exit code 70 and a line saying a script closed, shut down or replaced fd 3. `init` no longer takes `log_path` or `tail_chars`, so the runtime writes no journal for a background script and keeps no tail: the host writes both from fds 1 and 2, where a background script's stderr joins its stdout. **Unchanged by this change:** `PyRunner`'s interface, `ExecResult` included, and a `pythonPath` that names an interpreter or a wrapper that `exec`s one, of which it needs nothing more. The interpreter's new working directory, where a wrapper's own relative paths now resolve, comes from #193, whose entry says what it changes. No sibling package needs a new version. diff --git a/changelog.d/fix-code-execution-output-to-host.fixed.md b/changelog.d/fix-code-execution-output-to-host.fixed.md new file mode 100644 index 000000000..d17320f3a --- /dev/null +++ b/changelog.d/fix-code-execution-output-to-host.fixed.md @@ -0,0 +1,8 @@ +- What the commands a `code_execution` script runs write to stdout and stderr now comes back in the script's result, beside the script's own prints, line by line in the order written on each stream: `subprocess` without `capture_output`, `os.system`, and C code writing to file descriptors 1 and 2. It used to reach only the host's log, and a line a command wrote without a newline ran into the runtime's own reply, so the script waited out its time limit. A command with a lot of output now fills the result as the script's own prints do: each stream keeps up to 2,000,000 characters, counted as UTF-16 code units, so a character outside the Basic Multilingual Plane, such as an emoji, counts as two, and a long result spills to a workspace file, as any result does. Output belongs to the script running when it arrives: a command a script left running writes into the result of a later script running then, and into the host's log between scripts or once its interpreter has been replaced. The tool's description now says that what the script and the commands it runs print comes back, and that a background script's output streams to a log file when the agent has a writable workspace. That changes the tool definitions every request carries, and by [Anthropic's prompt-caching documentation](https://platform.claude.com/docs/en/build-with-claude/prompt-caching#what-invalidates-the-cache), a prompt cache written before the upgrade isn't reused. +- A command a `code_execution` script runs now reads the end of its input at once: its standard input is `/dev/null`. It used to read the runtime's channel from the host, so a command that reads its input, such as `cat`, or `ssh` without `-n`, took the host's messages, and the script failed or waited out its time limit. +- A script that closes or redirects its file descriptors 0, 1 or 2 leaves the next script its output: the runtime's channel to the host has file descriptors of its own, 3 and 4, which no command a script runs is given, and each script starts with 0, 1 and 2 as the host set them, blocking even after a command set one non-blocking. A script that points 1 or 2 at the runtime's channel gets it put back at its end, so a partial line it left there reaches its result; what a script or its commands write onto that channel, or read from it, can still garble the runtime's messages: the script's own, and, from a command started holding the channel, a later script's for as long as that command runs. A script whose result is garbled that way waits out its time limit and the grace, and the interpreter restarts, so the next script's variables are gone. A script's `os.close(1)` used to leave it waiting out its time limit, and its `os.close(0)` sometimes left the next script doing so. A script that closes or replaces one of the runtime's own file descriptors, 3, 4, or its copies of 0 to 2, now ends the interpreter by the end of that script at the latest, and the result says which and what to check, as does one that shuts down the socket under file descriptor 1, 3 or 4. That line is lost when the script also closed or replaced both the runtime's copy of file descriptor 2 and file descriptor 2 itself, or shut down the socket under them, and the result leaves it out once the script's stderr has passed its 2,000,000-character cap. +- What a script and its commands wrote before it was stopped, as far as it had reached the host, now comes back ahead of the message saying why, when it's killed at its time limit, aborted, or its interpreter crashes; it used to be lost. A partial line still in the script's writer when the interpreter is killed or crashes is lost with it. A crash's own text comes back too, and goes to the host's log: a fatal error's, which goes to the script's file descriptor 2 wherever the script pointed it, and now a traceback for a segfault or an abort, which faulthandler writes where CPython alone wrote nothing. +- A background script's journal now holds its commands' output too, each line as it arrives, and ends saying why the host stopped the script, when it did, a time limit's note included, unless the journal reached its 50 MB cap, now counted in bytes, as its cap line says, or a write to it failed. A background script that crashes leaves the last of what it wrote, its crash's text included and after it, in the message that wakes its agent, where that held only the message about the exit. A journal that can't be opened or written says so once in the host's log. A background script started where no workspace is writable is told that `wake_agent()` reaches its agent and, if the script dies, the last of its output reaches it too, where it was told its output wasn't retrievable. +- A script that prints a lone surrogate, such as `"\udcff"`, now gets its output back with it written as `\udcff`, rather than as U+FFFD. +- A script's prints now reach the host as each line is complete, so a partial line waits for its newline, a flush or the end of the script, as Python's stdout does on a terminal, and what a wrapper of the script's own in `sys.stdout` or `sys.stderr` holds is flushed at the script's end, as Python's exit would. Printing costs more than it did: 100,000 short `print` calls took about 0.36 s, against about 0.15 s before, on Linux with Node 24.22.0 and Python 3.13.5. +- An interpreter that exits as it starts now says in the result what it wrote to stderr, up to its last 2,000 characters, with "…" where they're cut, and one ended by a signal as it starts names the signal. Output that arrives while its interpreter runs no script, such as from a command a script left running, even once that interpreter has ended, reaches the host's log as `stdout while its interpreter runs no script: …` or `stderr while its interpreter runs no script: …`, each piece cut at 2,000 characters, where stdout's used to read `non-protocol stdout line: …`: up to 65,536 characters of such lines, counted as UTF-16 code units, on each pipe from the start of its interpreter's last script, then a line saying the host drops the rest while the interpreter runs no script. While the interpreter lives, the host reads such output as it's written; once the host has torn the interpreter down, as at an idle reclaim, it reads it at 1 MiB a second at most, so a command left running that writes without end slows to that rather than costing the host a core, and keeps running. A host that turns the idle reclaim off reads such a command at full speed for as long as its interpreter lives. When the host's messages stop reaching an interpreter, the host now gives it a moment to exit on its own before killing it, so one that died, as by running out of memory, reads as its own exit rather than as one that stopped reading; and the stopped-reading message no longer says a script closed its standard input: it says the interpreter stopped reading the host's messages, and to check whether the script closes, shuts down or replaces file descriptor 3. diff --git a/src/code-execution/py-runner.ts b/src/code-execution/py-runner.ts index 5da51ff1c..0fe6c2f1b 100644 --- a/src/code-execution/py-runner.ts +++ b/src/code-execution/py-runner.ts @@ -12,14 +12,22 @@ * What this class guarantees is liveness: a wedged or runaway script cannot * hang the agent turn (cancel + SIGINT -> grace -> SIGKILL -> respawn) and a crashed * interpreter surfaces as a tool result, never as an unhandled rejection. + * + * The protocol runs on the interpreter's fds 3 (host -> py) and 4 (py -> host); + * its fd 0 is /dev/null, and the host reads its fds 1 and 2 itself (see + * OutputStream), so what a script and the commands it runs write to them + * reaches the host whatever then happens to the interpreter. */ -import { spawn, type ChildProcessWithoutNullStreams } from 'node:child_process'; +import { spawn, type ChildProcess } from 'node:child_process'; import { buildChildEnv } from '../mcpl/transport.js'; -import { mkdtempSync, rmSync, writeFileSync } from 'node:fs'; +import { randomBytes } from 'node:crypto'; +import { closeSync, mkdirSync, mkdtempSync, openSync, rmSync, writeFileSync, writeSync } from 'node:fs'; import { tmpdir } from 'node:os'; -import { join, resolve } from 'node:path'; +import { dirname, join, resolve } from 'node:path'; import { createInterface, type Interface } from 'node:readline'; +import type { Readable, Writable } from 'node:stream'; +import { StringDecoder } from 'node:string_decoder'; import { boundedErrorMessage } from '../error-message.js'; import { PYTHON_RUNTIME_SOURCE } from './runtime-py.js'; @@ -43,7 +51,7 @@ export interface ExecResult { /** Per-exec options for background (daemon) scripts. */ export interface BackgroundExecOptions { - /** Absolute path for the line-buffered stdout+stderr journal (null: tail only). */ + /** Absolute path for the stdout+stderr journal, which the runner appends to as the output arrives (null: tail only). */ logPath: string | null; /** * Called on each wake_agent() from the script. Resolve with null to ack @@ -81,6 +89,7 @@ export type ScriptToolCallHandler = ( ) => Promise; export interface PyRunnerOptions { + /** Python to spawn (default 'python3'). A wrapper must pass fds 3 and 4 on, where the protocol runs (see CodeExecutionConfig). */ pythonPath?: string; /** Per-tool-call timeout surfaced as TimeoutError inside the script. */ toolCallTimeoutMs?: number; @@ -114,6 +123,50 @@ const DEFAULT_IDLE_RECLAIM_MS = 300_000; const CANCEL_GRACE_MS = 10_000; /** After the deadline, SIGINT is repeated at this interval until the script reports or is killed. */ const INTERRUPT_REPEAT_MS = 200; +/** Each of a script's stdout and stderr is kept up to this many characters in its result. */ +const OUTPUT_CAP_CHARS = 2_000_000; +/** A background script's journal stops growing past this many bytes. */ +const LOG_FILE_CAP_BYTES = 50_000_000; +/** How much of a background script's output its result keeps, for the envelope that tells the agent how it ended. */ +const TAIL_CHARS = 4000; +/** + * How long an interpreter that exited unexpectedly gets for the last of what it + * wrote, its output and its messages, to arrive when something still holds its + * pipes open, such as a command its script left running; otherwise their end + * says it has all arrived. + */ +const EXIT_OUTPUT_GRACE_MS = 200; +/** How long a message failing to reach an interpreter waits for the interpreter's own exit before the host ends it as one that stopped reading. */ +const UNREACHABLE_GRACE_MS = 200; +/** How much of what an interpreter wrote before it failed to start, or crashed, goes into the message and the host's log, and how much of each piece of output with no script running goes into the log. */ +const LAST_WORDS_CHARS = 2000; +/** + * How much of the host's log a pipe's output takes while no script of its + * interpreter runs, from the start of its last exec, or its own start: lines + * of up to 65,536 characters in all, counted as UTF-16 code units, each piece + * cut at LAST_WORDS_CHARS, enough to see what a command a script left running + * is doing. The rest is dropped. + */ +const IDLE_LOG_CHARS = 65_536; +/** + * How fast the host reads the pipes of an interpreter it has torn down, which + * a command a script left running may still hold. It goes on reading, so such + * a command isn't ended by a broken pipe, and one writing far under this, as a + * dev server's log does, never waits; one that writes without end slows to + * this, rather than costing the host a core for as long as it runs. A live + * interpreter's pipes are read at full speed, so what such a command writes + * between scripts is read as it's written, and the idle reclaim bounds how + * long that lasts, unless a host turns the reclaim off. + */ +const IDLE_READ_BYTES_PER_SECOND = 1_048_576; +/** The runtime's exit when the host's channel, fds 3 and 4, didn't reach it: the host's configuration, which calling again doesn't change. 78 is sysexits' EX_CONFIG, which Python itself never exits with. */ +const HOST_CHANNEL_MISSING_EXIT = 78; +/** + * The runtime's exit when it ends itself (runtime-py.ts _fatal), after a line + * that, when a script took one of its fds, says which. Read with that line, so + * a script that prints such a line and ends any other way isn't taken for it. + */ +const RUNTIME_FATAL_EXIT = 70; /** The longest delay Node's timers honour (~24.8 days); a longer one fires after ~1 ms. */ export const MAX_TIMER_MS = 2_147_483_647; @@ -125,6 +178,348 @@ export function formatLimit(ms: number): string { return `${Math.round(ms / 360_000) / 10} h`; } +/** `line` after `text`, on a line of its own: a text that doesn't end with a newline gets one first. */ +function onItsOwnLine(text: string, line: string): string { + return (text === '' || text.endsWith('\n') ? text : `${text}\n`) + line; +} + +/** Add a line of the runner's own to a background script's journal, on a line of its own. */ +function writeLine(journal: Journal, line: string): void { + journal.write(`${journal.tail === '' || journal.tail.endsWith('\n') ? '' : '\n'}${line}\n`); +} + +/** Whether the child is still running: not exited, not ended by a signal, and not killed by us. */ +function alive(child: ChildProcess): boolean { + return child.exitCode === null && child.signalCode === null && !child.killed; +} + +/** + * All of `text` at the end of the file: a write the kernel cuts short, as at a + * file-size limit, goes on from where it stopped, and one that fails throws. + */ +function writeAll(fd: number, text: string): void { + const data = Buffer.from(text); + for (let at = 0; at < data.length; ) { + const n = writeSync(fd, data, at); + // POSIX has a regular file take at least a byte of a write or fail it; a + // write that took none would otherwise hold the host's event loop here. + if (n === 0) throw new Error('the file took none of a write'); + at += n; + } +} + +/** + * A background script's journal: the log file the agent can read with its own + * tools while the script runs, and the tail its result carries. It's written + * synchronously, from the host's event loop, so a workspace whose writes block + * holds the host while they do. + */ +class Journal { + private fd: number | null = null; + private written = 0; + private capped = false; + tail = ''; + + constructor( + path: string | null, + private readonly log: (text: string) => void, + ) { + if (!path) return; + try { + mkdirSync(dirname(path), { recursive: true }); + this.fd = openSync(path, 'a'); + } catch (err) { + log(`cannot open log file: ${err instanceof Error ? err.message : String(err)}`); + } + } + + write(text: string): void { + if (text === '') return; + this.tail = (this.tail + text).slice(-TAIL_CHARS); + if (this.fd === null || this.capped) return; + const bytes = Buffer.byteLength(text); + try { + if (this.written + bytes > LOG_FILE_CAP_BYTES) { + this.capped = true; + writeAll(this.fd, `\n[log capped at ${LOG_FILE_CAP_BYTES} bytes]\n`); + } else { + writeAll(this.fd, text); + this.written += bytes; + } + } catch (err) { + // A journal that can't be written to still keeps its tail. + this.log(`cannot write the log file any more: ${err instanceof Error ? err.message : String(err)}`); + this.close(); + } + } + + close(): void { + if (this.fd === null) return; + try { + closeSync(this.fd); + } catch { + // already gone + } + this.fd = null; + } +} + +const NOTHING = Buffer.alloc(0); + +/** The first n UTF-16 code units of s, short of a surrogate pair the cut would split. */ +function cutWhole(s: string, n: number): string { + if (s.length <= n) return s; + const cut = s.slice(0, n); + return /[\uD800-\uDBFF]$/.test(cut) ? cut.slice(0, -1) : cut; +} + +/** + * One of the interpreter's output pipes, its fd 1 or fd 2, as the host reads it. + * While an exec runs, what arrives before that exec's end mark is the exec's: + * kept for its result, up to OUTPUT_CAP_CHARS, or written to a background + * script's journal. The mark itself is cut out. What arrives while no exec of + * its interpreter runs, such as what a command a script left running writes, + * goes to the host's log as it arrives, both while the interpreter lives and + * after it has ended, up to IDLE_LOG_CHARS between scripts; once the runner + * has torn the interpreter down, it's read at IDLE_READ_BYTES_PER_SECOND at + * most (retire). + */ +export class OutputStream { + /** Every process holding the pipe has closed it, so nothing more will come. */ + ended = false; + /** The running exec's mark has arrived, so all of its output on this pipe has. */ + marked = false; + /** The end of what arrived while no exec ran: an interpreter that failed to start says why here. */ + lastIdle = ''; + /** Whether lastIdle lost its beginning to the LAST_WORDS_CHARS cut. */ + lastIdleCut = false; + private mark: Buffer | null = null; + /** The end of a chunk that could be the start of a mark the next chunk completes. */ + private carry: Buffer = NOTHING; + private readonly decoder = new StringDecoder('utf8'); + private text = ''; + private truncated = false; + private journal: Journal | null = null; + private onEnd: (() => void)[] = []; + /** How many characters of log lines output with no script running has taken since this pipe's last exec began: IDLE_LOG_CHARS at most. */ + private idleLogged = 0; + /** Set once the runner has torn its interpreter down: nothing reads this pipe for an exec again. */ + private retired = false; + /** The wait that holds a retired pipe back (slow). */ + private resumer: ReturnType | null = null; + /** Bytes read since the last wait was set, which the next one charges (slow). */ + private owed = 0; + + constructor( + private readonly name: 'stdout' | 'stderr', + private readonly pipe: Readable, + private readonly log: (text: string) => void, + private readonly onMarked: () => void, + ) { + pipe.on('data', (chunk: Buffer) => this.push(chunk)); + const ended = () => { + if (this.ended) return; + this.ended = true; + for (const fn of this.onEnd.splice(0)) fn(); + }; + pipe.on('end', ended); + pipe.on('close', ended); + pipe.on('error', (err) => { + log(`reading the python runtime's ${name} failed: ${err.message}`); + ended(); + }); + } + + /** Begin an exec's share of the pipe, which its mark ends. */ + begin(mark: Buffer, journal: Journal | null): void { + // A character that output with no script running left half-written is + // that output's, not this exec's. + this.logIdle(this.decoder.end()); + // The stretch with no script running that follows this exec logs a share + // of its own. + this.idleLogged = 0; + this.mark = mark; + this.marked = false; + this.journal = journal; + this.text = ''; + this.truncated = false; + } + + /** The exec's share as it stands, with a note if it was cut at the cap; what arrives next goes to the log. */ + take(): string { + if (this.mark !== null) { + // Settled before its mark came (a kill, an abort or a crash): what arrived counts. + this.keep(this.carry); + this.carry = NOTHING; + this.keep(this.decoder.end()); + } + const text = this.text + (this.truncated ? `\n[${this.name} truncated at ${OUTPUT_CAP_CHARS} chars]` : ''); + this.mark = null; + this.journal = null; + this.text = ''; + this.truncated = false; + return text; + } + + /** Run fn once the pipe has ended (at once if it has). */ + whenEnded(fn: () => void): void { + if (this.ended) fn(); + else this.onEnd.push(fn); + } + + private push(chunk: Buffer): void { + let bytes = this.carry.length > 0 ? Buffer.concat([this.carry, chunk]) : chunk; + this.carry = NOTHING; + if (this.mark !== null) { + const at = bytes.indexOf(this.mark); + if (at < 0) { + // Hold back only an end that could be the mark's start, so a + // background script's journal keeps up with its output. + const hold = markStartAtEnd(bytes, this.mark); + this.keep(bytes.subarray(0, bytes.length - hold)); + this.carry = bytes.subarray(bytes.length - hold); + return; + } + this.keep(bytes.subarray(0, at)); + this.keep(this.decoder.end()); + bytes = bytes.subarray(at + this.mark.length); + this.mark = null; + this.marked = true; + this.onMarked(); + } + if (bytes.length > 0) { + const text = this.decoder.write(bytes); + const idle = this.lastIdle + text; + if (idle.length > LAST_WORDS_CHARS) { + this.lastIdleCut = true; + // A cut between a surrogate pair's halves would leave its low half, which reads as U+FFFD. + const kept = idle.slice(-LAST_WORDS_CHARS); + this.lastIdle = /^[\uDC00-\uDFFF]/.test(kept) ? kept.slice(1) : kept; + } else { + this.lastIdle = idle; + } + this.logIdle(text); + if (this.retired) this.slow(bytes.length); + } + } + + /** The runner has torn its interpreter down: read what still comes at IDLE_READ_BYTES_PER_SECOND at most. */ + retire(): void { + this.retired = true; + } + + /** + * Log output that arrived while no exec of this pipe's interpreter ran, each + * piece up to LAST_WORDS_CHARS, in lines of IDLE_LOG_CHARS in all since the + * pipe's last exec began, the last cut to fit. Each line is charged its + * whole length, so many small writes can't take more of the log. + */ + private logIdle(text: string): void { + const shown = text.trimEnd(); + if (this.idleLogged >= IDLE_LOG_CHARS || shown.trim() === '') return; + const head = `${this.name} while its interpreter runs no script: `; + const kept = cutWhole(shown, LAST_WORDS_CHARS); + const piece = kept.length < shown.length ? `${kept} [… ${shown.length - kept.length} more characters]` : shown; + const fits = IDLE_LOG_CHARS - this.idleLogged - head.length; + if (fits > 0) { + const line = head + cutWhole(piece, fits); + this.idleLogged += line.length; + this.log(line); + } + if (fits <= piece.length) { + this.idleLogged = IDLE_LOG_CHARS; + this.log(`${head}past ${IDLE_LOG_CHARS} characters of the log, so the host drops what follows while its interpreter runs no script`); + } + } + + /** + * Hold the pipe back for as long as these bytes take at + * IDLE_READ_BYTES_PER_SECOND, then read on. Something else can resume the + * pipe during a wait, as Node does with a child's stdio when the child + * exits, so each piece holds it back again, and what came during a wait + * waits its own time once that wait ends. + */ + private slow(bytes: number): void { + this.pipe.pause(); + this.owed += bytes; + if (this.resumer === null) this.wait(); + } + + private wait(): void { + const ms = Math.ceil((this.owed * 1000) / IDLE_READ_BYTES_PER_SECOND); + this.owed = 0; + this.resumer = setTimeout(() => { + this.resumer = null; + if (this.owed > 0) this.wait(); + else this.pipe.resume(); + }, ms); + // A command left running doesn't keep the host's process alive. + this.resumer.unref(); + } + + /** Keep what arrived for the running exec. */ + private keep(part: Buffer | string): void { + const text = typeof part === 'string' ? part : this.decoder.write(part); + if (text === '') return; + if (this.journal) { + this.journal.write(text); + return; + } + if (this.truncated) return; + const room = OUTPUT_CAP_CHARS - this.text.length; + if (text.length <= room) { + this.text += text; + return; + } + this.text += text.slice(0, room); + this.truncated = true; + } +} + +/** How many bytes at the end of `bytes` could be the start of `mark`. */ +function markStartAtEnd(bytes: Buffer, mark: Buffer): number { + for (let n = Math.min(bytes.length, mark.length - 1); n > 0; n--) { + if (bytes.subarray(bytes.length - n).equals(mark.subarray(0, n))) return n; + } + return 0; +} + +/** Run fn once the readable has ended (at once if it has). */ +function whenReadableEnded(stream: Readable, fn: () => void): void { + if (stream.readableEnded || stream.destroyed) { + fn(); + return; + } + let called = false; + const once = () => { + if (called) return; + called = true; + fn(); + }; + stream.once('end', once); + stream.once('close', once); +} + +/** + * Run fn once an exited interpreter's output pipes and its messages' pipe have + * all ended, which says the last of what it wrote has arrived, or else after + * EXIT_OUTPUT_GRACE_MS, if something, such as a command its script left + * running, still holds them, and one more round of the event loop's reads, so + * what had already arrived is read first even when the host's loop was busy + * past the grace. + */ +function afterOutputEnds(stdout: OutputStream, stderr: OutputStream, messages: Readable | null, fn: () => void): void { + let done = false; + const finish = () => { + if (done) return; + done = true; + clearTimeout(timer); + fn(); + }; + const timer = setTimeout(() => setImmediate(finish), EXIT_OUTPUT_GRACE_MS); + stdout.whenEnded(() => stderr.whenEnded(() => (messages ? whenReadableEnded(messages, finish) : finish()))); +} + /** * What a script stopped at its time limit is told after why: that the tool * calls it made keep their effects, and whether a longer limit can help, @@ -150,10 +545,16 @@ function limitRemedy(limitMs: number, ceilingMs: number | undefined, background: * host changes, so that one says code_execution can't run here rather than * to try again. A failure before the spawn, while creating or writing the * runtime's files, is about those files rather than the interpreter, and - * keeps Node's own text, which names the path for an open or a mkdtemp. + * keeps Node's own text, which names the path for an open or a mkdtemp. An + * interpreter that exits as it starts passes on up to the last LAST_WORDS_CHARS + * of what it wrote to stderr, from the end, where a traceback says what failed, + * and with "…" where they're cut; when the runtime itself exits so because + * the host's channel didn't reach it (HOST_CHANNEL_MISSING_EXIT), its own + * words say who must act and that calling again fails the same way, so they + * end the text. */ function startFailure(pythonPath: string, err: unknown): string { - const { code, syscall } = (err ?? {}) as { code?: unknown; syscall?: unknown }; + const { code, syscall, exitCode, said } = (err ?? {}) as { code?: unknown; syscall?: unknown; exitCode?: unknown; said?: unknown }; const spawning = typeof syscall === 'string' && syscall.startsWith('spawn'); if (spawning && (code === 'ENOENT' || code === 'EACCES')) { const until = code === 'ENOENT' ? 'installs it or points this host at one' : 'makes it executable or points this host at another'; @@ -162,17 +563,56 @@ function startFailure(pythonPath: string, err: unknown): string { if (typeof syscall === 'string' && !spawning) { return `script didn't run: this host couldn't create or write the files its Python runtime starts from (${boundedErrorMessage(err)}), so none of the script ran. Run it again; if it fails the same way, ask whoever runs this host.`; } + const wrote = typeof said === 'string' ? said : ''; + // The runtime's own words say who must act, and that calling again fails the same way. + if (exitCode === HOST_CHANNEL_MISSING_EXIT && wrote !== '') { + return `script didn't run: the Python interpreter, ${pythonPath}, exited as it started, saying: ${wrote}`; + } const why = typeof code === 'string' ? `couldn't be started (${code})` : boundedErrorMessage(err); - return `script didn't run: the Python interpreter, ${pythonPath}, ${why}, so none of the script ran. Run it again; if it fails the same way, ask whoever runs this host.`; + return `script didn't run: the Python interpreter, ${pythonPath}, ${why}, so none of the script ran. Run it again; if it fails the same way, ask whoever runs this host.` + + (wrote === '' ? '' : ` It wrote to stderr: ${wrote}`); } -/** Why a script stopped when its interpreter exited under it, and what running it again does. */ -function interpreterExit(exitCode: number | null, signal: NodeJS.Signals | null, background: boolean): string { - const how = signal ? `was ended by ${signal}` : `exited with code ${exitCode}`; - return `script stopped: the Python interpreter ${how} while it ran, so the rest of the script didn't run` +/** Why a script stopped when its interpreter stopped reading the host's messages, so that the host ended it. */ +function stoppedReading(background: boolean): string { + return "script stopped: the Python interpreter stopped reading the host's messages, so the host ended it and the rest of the script didn't run" + `${background ? '' : ', and variables from earlier scripts are gone'}. Tool calls it had already made keep their effects, ` - + 'so check them before you run it again in a new interpreter. If it stops the same way, check whether the script itself ' - + `ends the process, for example ${signal ? 'by using too much memory' : 'with os._exit()'}; if not, ask whoever runs this host.`; + + 'so check them before you run it again in a new interpreter. If it stops the same way, check whether the script closes, ' + + "shuts down or replaces file descriptor 3, where the host's messages arrive; if not, ask whoever runs this host."; +} + +/** + * Why a script stopped when its interpreter exited under it, and what running it + * again does; or, when the script's result came before the exit (ended), that + * the script ran to its end and what of its output the exit may have cost. + */ +function interpreterExit(exitCode: number | null, signal: NodeJS.Signals | null, background: boolean, said = '', ended = false): string { + const how = signal ? `was ended by ${signal}` : `exited with code ${exitCode}`; + // The runtime ends itself (_fatal) with code 70 after a line saying which of its fds a script took, + // or shut down the socket under. The line reaches the host unless the script took both the runtime's + // copy of fd 2 and fd 2 itself, or shut down the socket under them, and it's read from what the result + // keeps, which past a stream's cap or a journal's tail lacks it. + const taken = exitCode === RUNTIME_FATAL_EXIT ? fdTakenFromRuntime(said) : null; + const check = taken + ? `check whether the script closes, shuts down or replaces ${taken}, as the runtime's line above says` + : `check whether the script itself ends the process, for example ${signal ? 'by using too much memory' : 'with os._exit()'}`; + const gone = background ? '' : ', and variables from earlier scripts are gone'; + if (ended) { + return `script ended, but the Python interpreter ${how} before all of its output reached the host, so some of it may be missing${gone}. ` + + `If it happens again, ${check}; if not, ask whoever runs this host.`; + } + return `script stopped: the Python interpreter ${how} while it ran, so the rest of the script didn't run${gone}. ` + + 'Tool calls it had already made keep their effects, so check them before you run it again in a new interpreter. ' + + `If it stops the same way, ${check}; if not, ask whoever runs this host.`; +} + +/** The file descriptor the runtime's last line of its own says a script took from it (runtime-py.ts _lost), if it says so. */ +function fdTakenFromRuntime(said: string): string | null { + let taken: string | null = null; + for (const m of said.matchAll(/\[pytc-runtime\] [^\n]*: a script closed, shut down or replaced ((?:the runtime's copy of )?fd \d)[^\n]*; exiting/g)) { + taken = m[1]; + } + return taken; } interface PendingExec { @@ -195,6 +635,13 @@ interface PendingExec { * waiting for the interpreter to start, and an interpreter that exits then * failed to start (see exec). */ sent: boolean; + /** A background exec's journal, which its output goes to instead of its result. */ + journal: Journal | null; + /** The script's return code, once the interpreter has reported it; the exec + * settles when its output's end marks have arrived too. */ + returnCode: number | null; + /** Set when the host settled the exec itself (stopPending), rather than its result and marks. */ + stopped?: true; } /** Interpreters started in this process, by every runner: each one's number. */ @@ -222,9 +669,17 @@ export class PyRunner { private readonly envOptions: { env?: Record; inheritEnv?: boolean }; private onWake: ((line: number, payload: unknown) => Promise) | null = null; - private child: ChildProcessWithoutNullStreams | null = null; + private child: ChildProcess | null = null; private childReady: Promise | null = null; + /** The interpreter's fd 3, where the host's messages go. */ + private protoIn: Writable | null = null; + /** Ends the interpreter whose fd 3 a message couldn't reach (see ensureChild). */ + private unreachable: ((why: string) => void) | null = null; + /** Reads the interpreter's messages from its fd 4. */ private reader: Interface | null = null; + /** The interpreter's stdout and stderr, as the host reads them. */ + private out: OutputStream | null = null; + private err: OutputStream | null = null; private runtimeDir: string | null = null; private pending: PendingExec | null = null; private idleTimer: ReturnType | null = null; @@ -320,6 +775,8 @@ export class PyRunner { backgroundCeilingMs: opts?.backgroundCeilingMs, background: background !== undefined, sent: false, + journal: null, + returnCode: null, }; this.pending = pending; try { @@ -359,31 +816,31 @@ export class PyRunner { pending.interruptTimer = setInterval(() => this.interruptChild(), INTERRUPT_REPEAT_MS); pending.killTimer = setTimeout(() => { pending.deadlineMs = null; // this message already says why - this.settlePending({ - stdout: '', - stderr: - `script stopped: it reached its ${formatLimit(deadlineMs)} time limit and did not respond, so it was killed` + + this.stopPending( + `script stopped: it reached its ${formatLimit(deadlineMs)} time limit and did not respond, so it was killed` + (pending.background ? '. ' : ' and the interpreter restarted (variables from earlier scripts are gone). ') + limitRemedy(deadlineMs, pending.ceilingMs, pending.background, pending.backgroundCeilingMs), - returnCode: 1, - aborted: true, - }); + ); this.reclaim('deadline-kill'); }, this.cancelGraceMs); }, deadlineMs); // A day-scale background deadline must not hold the process open. if (background) pending.deadlineTimer.unref?.(); + // What the script writes, up to its mark, is this exec's: a background + // script's goes to its journal. + const mark = Buffer.from(`\x1epytc-end-${randomBytes(16).toString('hex')}\x1e`); + pending.journal = background ? new Journal(background.logPath, (text) => this.log(text)) : null; + this.out?.begin(mark, pending.journal); + this.err?.begin(mark, pending.journal); pending.sent = true; this.send({ op: 'init', tools: tools.map((t) => ({ py_name: t.pyName, tool_name: t.toolName })), call_timeout_s: Math.round(this.toolCallTimeoutMs / 1000), - ...(background - ? { background: true, log_path: background.logPath ?? null } - : {}), + ...(background ? { background: true } : {}), }); - this.send({ op: 'exec', id: execId, code }); + this.send({ op: 'exec', id: execId, code, mark: mark.toString('utf8') }); const result = await settled; @@ -403,33 +860,21 @@ export class PyRunner { abort(reason: string): void { if (!this.pending) return; this.send({ op: 'cancel', id: this.pending.id, reason }); - this.settlePending({ - stdout: '', - stderr: `script aborted by host: ${reason}`, - returnCode: 1, - aborted: true, - }); + this.stopPending(`script aborted by host: ${reason}`); this.reclaim(`abort: ${reason}`); } dispose(): void { if (this.disposed) return; this.disposed = true; - if (this.pending) { - this.settlePending({ - stdout: '', - stderr: 'code_execution runner disposed', - returnCode: 1, - aborted: true, - }); - } + this.stopPending('code_execution runner disposed'); this.reclaim('dispose'); } // ------------------------------------------------------------------------- private async ensureChild(): Promise { - if (this.child && this.child.exitCode === null && !this.child.killed) { + if (this.child && alive(this.child)) { return this.childReady ?? Promise.resolve(); } this.teardownChild(); @@ -441,41 +886,66 @@ export class PyRunner { writeFileSync(runtimePath, PYTHON_RUNTIME_SOURCE, 'utf8'); const child = spawn(spawnablePath(this.pythonPath), [runtimePath], { - stdio: ['pipe', 'pipe', 'pipe'], + // fd 0 is /dev/null, the host reads fds 1 and 2, and the protocol has 3 and 4. + stdio: ['ignore', 'pipe', 'pipe', 'pipe', 'pipe'], // The runtime directory, not the host's working directory: a script's // relative paths land here, and go at teardown, rather than beside the // host's .env and data. An absolute path still reaches the host's files. cwd: this.runtimeDir, - // Set last, over a declared env: the protocol needs both. It writes - // non-ASCII JSON on piped stdio, which Python before 3.15 on Windows - // encodes in the ANSI code page unless told otherwise, and a host's - // own PYTHONUTF8 no longer reaches the interpreter. + // Set last, over a declared env. The host reads fds 1 and 2 as UTF-8. + // The runtime's channel and a script's writers say UTF-8 themselves + // (runtime-py.ts), but other Python writing there takes its encoding and + // buffering from these: the interpreter's sys.stdout and sys.stderr + // between scripts, which a thread a script left running prints through, + // and any Python a script starts. So what it writes reaches the host as + // it's written, and in UTF-8, where Python before 3.15 on Windows writes + // a pipe in the ANSI code page unless told otherwise; a PYTHONUTF8 in + // the host process's own environment reaches the interpreter only with + // inheritEnv: true or declared in env. env: { ...buildChildEnv(this.envOptions), PYTHONUNBUFFERED: '1', PYTHONIOENCODING: 'utf-8' }, }); this.child = child; this.interpreterNumber = ++interpretersStarted; + const protoIn = child.stdio[3] as Writable; + const protoOut = child.stdio[4] as Readable; + this.protoIn = protoIn; + const stdout = new OutputStream('stdout', child.stdout as Readable, (text) => this.log(text), () => this.settleIfComplete(child)); + const stderr = new OutputStream('stderr', child.stderr as Readable, (text) => this.log(text), () => this.settleIfComplete(child)); + this.out = stdout; + this.err = stderr; + protoOut.on('error', (e) => this.log(`reading the python runtime's messages failed: ${e.message}`)); - child.stderr.setEncoding('utf8'); - child.stderr.on('data', (chunk: string) => { - // Outside an exec, python stderr is runtime diagnostics (crash - // tracebacks etc.) — surface them in host logs. - console.error(`[pytc:${this.label}] ${chunk.trimEnd()}`); - }); - - // A write to an interpreter that no longer reads its input fails on the - // stream after write() returns; with no listener, that error is thrown as - // an uncaught exception. Such an interpreter can go on running, as after a - // script closes file descriptor 0, and every later message to it is lost: - // its exec would wait out its time limit. So end it, and its exit settles - // the exec now. One already dead or killed is left to its own exit. + // An interpreter that no longer reads the host's messages, as after a + // script closes file descriptor 3, could go on running, and every later + // message to it is lost: its exec would wait out its time limit. So when + // a message can't reach it, it's ended, and its exit settles the exec. Such + // a write fails after write() returns: the pipe refuses it, or has already + // ended, its far end closed. Its error, which with no listener would be + // thrown as an uncaught exception, is heard here too. An interpreter that + // died on its own breaks the pipe the same way, as by dying with a message + // unread, and its exit can be seen after the break: so the kill waits + // UNREACHABLE_GRACE_MS and one more round of reads, and an interpreter that + // exited by then is left to its own exit, which says what happened. let inputFailed = false; - child.stdin.on('error', (err) => { - console.error(`[pytc:${this.label}] write to the python runtime failed: ${err.message}`); - if (child.exitCode === null && !child.killed && child.pid !== undefined) { - inputFailed = true; - child.kill('SIGKILL'); - } - }); + let killing = false; + const unreachable = (why: string) => { + // One the host has torn down already ended its pipe on purpose. + if (child !== this.child || killing) return; + this.log(`messages to the python runtime failed: ${why}`); + killing = true; + setTimeout( + () => + setImmediate(() => { + if (child === this.child && alive(child) && child.pid !== undefined) { + inputFailed = true; + child.kill('SIGKILL'); + } + }), + UNREACHABLE_GRACE_MS, + ); + }; + this.unreachable = unreachable; + protoIn.on('error', (e) => unreachable(e.message)); child.on('error', (err) => { // Spawn failure lands here (e.g. python3 missing); readiness rejects. @@ -485,51 +955,79 @@ export class PyRunner { child.on('exit', (exitCode, signal) => { // An exec still waiting for the interpreter is settled by exec, as a // runtime that failed to start. - if (this.pending?.sent) { - this.settlePending({ - stdout: '', - stderr: - inputFailed && signal === 'SIGKILL' - ? 'python runtime stopped reading its input, so it was killed and the interpreter restarted ' + - '(variables from earlier scripts are gone). A script that closes its standard input ' + - '(file descriptor 0) does this.' - : interpreterExit(exitCode, signal, this.pending.background), - returnCode: 1, - aborted: true, - }); - } - if (this.child === child) { - this.teardownChild(); + const pending = this.pending; + if (!pending?.sent) { + // One still starting is reported by its exec, as a failed start. + if (!pending && this.child === child) { + this.log(`python runtime exited while no script ran (code=${exitCode}, signal=${signal ?? 'none'}); the next script starts a new one`); + } + if (this.child === child) this.teardownChild(); + return; } + const killedDeaf = inputFailed && signal === 'SIGKILL'; + // Why, from what arrived: the runtime's own exit names the fd it lost in its last line. + const why = (said: string) => + killedDeaf + ? stoppedReading(pending.background) + : interpreterExit(exitCode, signal, pending.background, said, pending.returnCode !== null); + // The host's log keeps the short form, with the exit's own code and signal. + const logged = killedDeaf + ? "python runtime stopped reading the host's messages, so it was killed" + : `python runtime exited unexpectedly (code=${exitCode}, signal=${signal ?? 'none'})`; + // What it wrote last, such as a fatal error's own text or an exec's + // result, is already on its way through its pipes: settle once they've + // ended, or after a short grace while a command it started still holds + // them. + afterOutputEnds(stdout, stderr, protoOut, () => { + if (this.pending === pending) { + const last = this.stopPending(why).trim(); + // A background script's last output is its journal's, both streams. + const what = pending.journal ? 'the last it wrote' : 'the last it wrote to stderr'; + this.log(last ? `${logged}; ${what}: ${last.slice(-LAST_WORDS_CHARS)}` : logged); + } else if (!pending.stopped) { + // Its script's result and marks, already on their way, settled the + // exec while the host waited: it exited as its script ended, whether + // or not a next exec has replaced it since. + this.log(`python runtime exited as its script ended (code=${exitCode}, signal=${signal ?? 'none'}); the next script starts a new one`); + } + if (this.child === child) this.teardownChild(); + }); }); - this.reader = createInterface({ input: child.stdout }); + this.reader = createInterface({ input: protoOut }); this.reader.on('line', (line) => this.handleLine(line, child)); + // readline passes its input's error on to the interface, where with no + // listener it would be thrown as an uncaught exception and end the host. + // protoOut's own listener above logs it. + this.reader.on('error', () => {}); this.childReady = new Promise((resolve, reject) => { - const onReady = () => { - cleanup(); - resolve(); - }; - const onExit = () => { - cleanup(); - reject(new Error('exited as it started')); - }; - const onError = (err: Error) => { - cleanup(); - reject(err); - }; - const timeout = setTimeout(() => { - cleanup(); - reject(new Error("didn't start within 15 seconds")); - }, 15_000); - const cleanup = () => { + let over = false; + const finish = (outcome: () => void) => { + if (over) return; + over = true; clearTimeout(timeout); - this.readyResolver = null; - this.readyFailer = null; + // Another interpreter may have started since this one exited. + if (this.readyResolver === onReady) this.readyResolver = null; + if (this.readyFailer === onError) this.readyFailer = null; child.off('exit', onExit); child.off('error', onError); + outcome(); }; + const onReady = () => finish(resolve); + const onExit = (exitCode: number | null, signal: NodeJS.Signals | null) => { + // Why it ended, such as a traceback, is on its way through its stderr. + // Its messages' pipe doesn't matter here, and the exit's teardown has + // stopped reading it. A tail that lost its beginning says so. + afterOutputEnds(stdout, stderr, null, () => { + const tail = stderr.lastIdle.trim(); + const said = tail !== '' && stderr.lastIdleCut ? `…${tail}` : tail; + const how = signal ? `was ended by ${signal} as it started` : 'exited as it started'; + finish(() => reject(Object.assign(new Error(how), { exitCode, said }))); + }); + }; + const onError = (e: Error) => finish(() => reject(e)); + const timeout = setTimeout(() => finish(() => reject(new Error("didn't start within 15 seconds"))), 15_000); this.readyResolver = onReady; this.readyFailer = onError; child.on('exit', onExit); @@ -538,16 +1036,20 @@ export class PyRunner { return this.childReady; } + private log(text: string): void { + console.error(`[pytc:${this.label}] ${text}`); + } + private readyResolver: (() => void) | null = null; /** Fails the readiness of an interpreter that is still starting (see reclaim). */ private readyFailer: ((err: Error) => void) | null = null; - private handleLine(line: string, child: ChildProcessWithoutNullStreams): void { - let msg: { op?: string; id?: string; name?: string; args?: unknown; stdout?: string; stderr?: string; return_code?: number }; + private handleLine(line: string, child: ChildProcess): void { + let msg: { op?: string; id?: string; name?: string; args?: unknown; return_code?: number }; try { msg = JSON.parse(line); } catch { - console.error(`[pytc:${this.label}] non-protocol stdout line: ${line.slice(0, 200)}`); + this.log(`non-protocol line from the python runtime: ${line.slice(0, 200)}`); return; } @@ -606,24 +1108,70 @@ export class PyRunner { } case 'exec_result': { - if (this.pending && msg.id === this.pending.id) { - this.settlePending({ - stdout: msg.stdout ?? '', - stderr: msg.stderr ?? '', - returnCode: typeof msg.return_code === 'number' ? msg.return_code : 1, - ...(typeof (msg as { tail?: unknown }).tail === 'string' - ? { tail: (msg as { tail: string }).tail } - : {}), - }); + const pending = this.pending; + if (pending && msg.id === pending.id && pending.returnCode === null) { + pending.returnCode = typeof msg.return_code === 'number' ? msg.return_code : 1; + this.settleIfComplete(child); } return; } default: - console.error(`[pytc:${this.label}] unknown protocol op: ${String(msg.op)}`); + this.log(`unknown protocol op: ${String(msg.op)}`); + } + } + + /** + * Settle the running exec once its script has reported and the end marks + * of both its output pipes have arrived, which says all of its output has. + */ + private settleIfComplete(child: ChildProcess): void { + const pending = this.pending; + if (child !== this.child || !pending || pending.returnCode === null) return; + if (!this.out?.marked || !this.err?.marked) return; + const stdout = this.out.take(); + const stderr = this.err.take(); + const journal = pending.journal; + if (journal) { + this.settlePending({ stdout: '', stderr: '', returnCode: pending.returnCode, tail: journal.tail }); + } else { + this.settlePending({ stdout, stderr, returnCode: pending.returnCode }); } } + /** + * Settle the running exec now: what its script wrote so far, as far as it + * has reached the host, then `why` it stopped, on a line of its own. Output + * still in the pipes goes to the host's log. A script that reported before + * its interpreter ended keeps its own return code. Returns what the script + * wrote to stderr (a background script's tail), for the host's log. + */ + private stopPending(reason: string | ((said: string) => string)): string { + const pending = this.pending; + if (!pending || pending.settled) return ''; + pending.stopped = true; + const stdout = this.out?.take() ?? ''; + const stderr = this.err?.take() ?? ''; + const returnCode = pending.returnCode ?? 1; + const journal = pending.journal; + // The runtime's own lines are in a background script's journal. A foreground + // script's are in its stderr, unless it pointed fd 2 at stdout's pipe and took + // the runtime's copy of fd 2 (runtime-py.ts _diag): stderr goes last, as the + // one the runtime writes to first, since the last line that names a fd counts. + const why = typeof reason === 'string' ? reason : reason(journal ? journal.tail : `${stdout}\n${stderr}`); + if (journal) { + const tail = journal.tail; + // The journal says why too: in its file, which the agent reads with its + // own tools, when it has one that hasn't reached its cap or failed a + // write, and in its tail, which the result carries. + writeLine(journal, why); + this.settlePending({ stdout: '', stderr: why, returnCode, aborted: true, tail: journal.tail }); + return tail; + } + this.settlePending({ stdout, stderr: onItsOwnLine(stderr, why), returnCode, aborted: true }); + return stderr; + } + private settlePending(result: ExecResult): void { const pending = this.pending; if (!pending || pending.settled) return; @@ -634,11 +1182,18 @@ export class PyRunner { if (pending.deadlineMs !== null && result.returnCode !== 0) { const note = `script stopped: it reached its ${formatLimit(pending.deadlineMs)} time limit. ` + `${limitRemedy(pending.deadlineMs, pending.ceilingMs, pending.background, pending.backgroundCeilingMs)}\n`; - // On a line of its own: the runner's own texts, such as an interpreter's exit, end without a newline. - const withNote = (text: string) => (text === '' || text.endsWith('\n') ? text : `${text}\n`) + note; - // A background script reports its output tail, so the note goes there too. - result = { ...result, stderr: withNote(result.stderr), ...(result.tail !== undefined ? { tail: withNote(result.tail) } : {}) }; + // On a line of its own: a script's output, and the runner's own texts, such as an + // interpreter's exit, can end without a newline. + result = { ...result, stderr: onItsOwnLine(result.stderr, note) }; + // A background script's journal says so too: in its file, when it has one, + // and in its tail, which is the result's. + if (pending.journal) { + writeLine(pending.journal, note.trimEnd()); + result = { ...result, tail: pending.journal.tail }; + } } + // Only now: what the stop wrote above is the journal's last. + pending.journal?.close(); if (pending.deadlineTimer) clearTimeout(pending.deadlineTimer); if (pending.killTimer) clearTimeout(pending.killTimer); if (pending.interruptTimer) clearInterval(pending.interruptTimer); @@ -652,9 +1207,9 @@ export class PyRunner { * aborted, killed or crashed, or the interpreter reclaimed) would resolve * the same id in the one that replaced it, inside another script: drop it. */ - private reply(child: ChildProcessWithoutNullStreams, msg: unknown, what: string): void { + private reply(child: ChildProcess, msg: unknown, what: string): void { if (child !== this.child) { - console.error(`[pytc:${this.label}] dropped late ${what}: the interpreter that asked for it is gone`); + this.log(`dropped late ${what}: the interpreter that asked for it is gone`); return; } this.send(msg); @@ -662,13 +1217,15 @@ export class PyRunner { private send(obj: unknown): void { const child = this.child; - if (!child || child.exitCode !== null || child.killed) return; + const input = this.protoIn; + const unreachable = this.unreachable; + if (!child || !input || !alive(child)) return; try { - child.stdin.write(JSON.stringify(obj) + '\n'); + input.write(JSON.stringify(obj) + '\n', (err) => { + if (err) unreachable?.(err.message); + }); } catch (err) { - console.error( - `[pytc:${this.label}] protocol write failed: ${err instanceof Error ? err.message : String(err)}`, - ); + this.log(`protocol write failed: ${err instanceof Error ? err.message : String(err)}`); } } @@ -682,7 +1239,7 @@ export class PyRunner { */ private interruptChild(): void { const child = this.child; - if (process.platform === 'win32' || !child || child.exitCode !== null || child.killed) return; + if (process.platform === 'win32' || !child || !alive(child)) return; if (child.pid === undefined) return; try { process.kill(child.pid, 'SIGINT'); @@ -694,7 +1251,7 @@ export class PyRunner { /** Kill interpreter + clean temp dir. State is lost (by design). */ private reclaim(reason: string): void { if (this.child) { - console.error(`[pytc:${this.label}] reclaiming python interpreter (${reason})`); + this.log(`reclaiming python interpreter (${reason})`); } // An interpreter killed as it starts never says it's ready: fail its // readiness now, so an exec waiting on it returns at once rather than at @@ -713,13 +1270,24 @@ export class PyRunner { } const child = this.child; if (child) { + const input = this.protoIn; this.child = null; this.childReady = null; this.readyResolver = null; this.readyFailer = null; + this.protoIn = null; + this.unreachable = null; + // Its output pipes stay read, into the host's log, until whatever still + // holds them, such as a command its script left running, closes them: + // breaking them under such a command could end it. They're read slower + // from now on, since nothing reads them for an exec again. + this.out?.retire(); + this.err?.retire(); + this.out = null; + this.err = null; child.removeAllListeners('exit'); try { - child.stdin.end(); + input?.end(); } catch { // stream may already be destroyed } @@ -728,7 +1296,7 @@ export class PyRunner { // from an abort or dispose on the tick exec was called, signals no // child of ours but whatever pid Node's handle held: the host's own // process group on Node 20.20.2, another process's pid on 24.19. - if (child.exitCode === null && !child.killed && child.pid !== undefined) { + if (alive(child) && child.pid !== undefined) { child.kill('SIGKILL'); } } diff --git a/src/code-execution/runtime-py.ts b/src/code-execution/runtime-py.ts index 89147ed35..967fc619a 100644 --- a/src/code-execution/runtime-py.ts +++ b/src/code-execution/runtime-py.ts @@ -20,23 +20,69 @@ * the payload to the host, which injects it into the agent's context and * triggers inference. Awaits a host ack so rate limiting backpressures * inside the script instead of dropping wakes. - * - stdout/stderr tee'd line-buffered to `log_path` (agent-readable via - * their file tools) with an in-memory tail kept for the crash envelope. + * - its stderr joined to its stdout (fd 2 onto fd 1's pipe), so the host's + * journal of its output (a log file the agent can read, and a tail for the + * crash envelope) holds their lines in the order they were written. The + * runtime's own lines and faulthandler's traceback go to that pipe too, so + * a crash's text comes after the lines the script wrote before it; a + * partial line still in a script's writer is lost with the process. + * + * OUTPUT. The protocol has file descriptors of its own: the host opens the + * runtime's fds 3 (host -> py) and 4 (py -> host), which the runtime makes + * non-inheritable as it starts, so no command a script runs is given them. fd 0 + * is /dev/null, and fds 1 and 2 are pipes the host reads itself: a script's + * prints, line by line, and the output of every command it runs go there in + * the order written, and what reaches them reaches the host whatever then + * happens to the interpreter. A partial line waits in the script's writer for + * its newline, a flush or the exec's end, as Python's stdout does on a + * terminal, and a command writing at the same moment as a long write can land + * between its parts. The host gives each exec what arrived on each stream + * before that exec's end mark (the exec op's `mark`), which the runtime writes + * into both streams once the script has returned; each stream is first in, + * first out, so what the script and the commands it waited for wrote is ahead + * of it. Nothing a script or a command does with fds 0, 1 or 2 reaches the + * protocol, short of pointing them at the runtime's channel, fds 3 and 4, + * which can garble its messages while the script runs and, through a command + * started holding the channel, for as long as that command runs. A garbled + * exec_result leaves its script waiting out its time limit and the grace, and + * the interpreter is restarted, so the next script's variables are gone. The + * runtime puts fds 0, 1 and 2 back as the host set them, blocking, at each + * exec's start, and puts back fd 1 or 2 pointed at the channel at the script's + * end too, before anything is flushed (_off_channel). Its own fds (3 and 4, and its copies of 0, 1 and 2) it + * checks at each exec's start and end, and fd 4 before each message: one a + * script closed or replaced ends the interpreter, saying which + * (_check_own_fds). So does a message or an end mark that can't be written, as + * after a script shuts down the socket under fd 4 or an output pipe, which + * leaves the fd as it was (send, _write_marks). * * Embedded as a TS string because tsc does not copy non-TS assets to dist/; * PyRunner materializes it to a temp file at spawn. * - * Protocol (JSON lines): + * Protocol (JSON lines, on fds 3 and 4): * host -> py : {op:"init", tools:[{py_name,tool_name}], call_timeout_s, - * background?, log_path?, tail_chars?} - * {op:"exec", id, code} + * background?} + * {op:"exec", id, code, mark} * {op:"tool_result", id, result, is_error?, structured_json?} * {op:"wake_ack", id, error?} * {op:"cancel", id, reason?} + * {op:"exit"} stops the runtime: a running script is + * cancelled at its next await, one blocked in + * synchronous code runs on until then (a host + * that can't wait sends SIGINT, as at a time + * limit), and it exits 0. It goes alone on a + * line under 100 characters, as {"op":"exit"} + * does: the reader thread looks for it only in + * such lines. A host that closes fd 3 instead + * while a script runs gets exit code 70 and the + * line "cannot read the host's messages any + * more: a script closed, shut down or replaced + * fd 3, where they arrive", since the runtime + * can't tell that from a script's own shutdown + * of the socket under fd 3. * py -> host : {op:"ready"} * {op:"tool_call", id, exec_id, name, args} * {op:"wake", id, exec_id, line, payload} - * {op:"exec_result", id, stdout, stderr, return_code, tail?} + * {op:"exec_result", id, return_code} * * At a deadline the host sends cancel and then SIGINT, repeated until the * script reports (not on Windows): the cancel stops a script waiting on an @@ -50,126 +96,434 @@ export const PYTHON_RUNTIME_SOURCE: string = String.raw` import ast import asyncio +import faulthandler import functools import inspect import io import json import os import re +import select import signal +import stat import sys import threading import traceback +import weakref -PROTO_OUT = sys.stdout # protocol channel (real stdout) -REAL_STDERR = sys.stderr # runtime diagnostics outside execs CALL_TIMEOUT_S = 270.0 -OUTPUT_CAP_CHARS = 2_000_000 -LOG_FILE_CAP_BYTES = 50_000_000 - BACKGROUND = False -LOG_PATH = None -TAIL_CHARS = 4000 + +# A process a script forks has the runtime's state too; at its script's end +# it leaves, since the exec and its result are the runtime's (_run_script). +_RUNTIME_PID = os.getpid() + +# The protocol's own channel, fds 3 and 4 as the host opened them (_start). +PROTO_IN = None +PROTO_OUT = None + +# Private copies of fds 0, 1 and 2 as the host set them up: /dev/null, and the +# two output pipes the host reads (_take_std_fds). Each exec puts fds 0, 1 and 2 +# back from them, whatever an earlier script did with its own, and writes its +# end marks through them. os.dup's copies are non-inheritable, so no command a +# script runs holds them. +_STD_IN = None +_STD_OUT = None +_STD_ERR = None + +# The runtime's own fds, as it found them: fd -> (what losing it stops, which +# fd it is, what it was). A script can close or replace any of them (os.close, +# os.dup2), so each exec's start and end checks them (_check_own_fds). +_OWN_FDS = {} + +# Every writer the runtime gave a script, which a script can keep past its +# exec (a logging handler keeps sys.stderr): each exec's end flushes them all, +# so a partial line goes out in the exec that wrote it. +_script_writers = weakref.WeakSet() _send_lock = threading.Lock() # A str can hold a lone surrogate, which UTF-8 can't encode: on POSIX, # os.fsdecode makes one of each byte in a filename that doesn't decode as -# UTF-8, and json.loads makes one of a "\ud800" escape. PYTHONIOENCODING makes -# the protocol channel strict UTF-8, so writing one would fail the whole -# message, and a script's result would never arrive. Each goes as U+FFFD. +# UTF-8, and json.loads makes one of a "\ud800" escape. The protocol's channel +# is strict UTF-8 (_start), so writing one would fail the whole message, and a +# tool call or a wake carrying one would never reach the host. Each goes as +# U+FFFD. What a script prints shows it escaped, as backslashreplace writes +# it (_script_writer). _LONE_SURROGATE = re.compile(r"[\ud800-\udfff]") def send(obj): line = _LONE_SURROGATE.sub("\ufffd", json.dumps(obj, ensure_ascii=False, default=str)) with _send_lock: - PROTO_OUT.write(line + "\n") - PROTO_OUT.flush() + # A script can close or replace fd 4 before a tool call, as well as + # before its end, so each message checks it first. + _check_own_fd(4) + try: + PROTO_OUT.write(line + "\n") + PROTO_OUT.flush() + except OSError as exc: + # fd 4 is still the host's, so the socket under it broke: a script + # shut it down, or the host's end is gone. Nothing more reaches it. + _fatal(_lost(4) + " (" + str(exc) + ")") + + +def _start(): + global PROTO_IN, PROTO_OUT + # A wrapper between the host and this interpreter can close fds 3 and 4 + # (a container, or sudo with its default settings, does): say so where the + # host reads, for a failed start that names what to fix, and stop with + # EX_CONFIG (78), which tells the host that calling again won't help + # (HOST_CHANNEL_MISSING_EXIT in py-runner.ts). + missing = [] + for fd in (3, 4): + try: + os.fstat(fd) + except OSError: + missing.append(fd) + if missing: + which = ("its file descriptors 3 and 4 aren't open" if len(missing) == 2 + else "its file descriptor " + str(missing[0]) + " isn't open") + try: + os.write(2, ( + "The host's channel didn't reach this Python interpreter: " + which + ", as when the " + "host's pythonPath starts Python through a wrapper that doesn't pass both file descriptors " + "3 and 4 on, such as a container, or sudo with its default settings. code_execution can't " + "run scripts here until whoever runs this host changes that, so calling it again fails " + "the same way.\n" + ).encode()) + except OSError: + pass + sys.exit(78) + # fds 3 and 4 came from the host inheritable; no command a script runs may + # hold them, or read the host's messages. + os.set_inheritable(3, False) + os.set_inheritable(4, False) + _own(3, "cannot read the host's messages any more", "fd 3, where they arrive") + _own(4, "cannot send the host a message any more", "fd 4, where they go") + PROTO_IN = open(3, "r", encoding="utf-8", newline="\n") + PROTO_OUT = open(4, "w", encoding="utf-8", newline="\n") + # Between scripts, a thread a script left running prints through these: + # a lone surrogate is escaped there, as in a script's own writers, rather + # than raising in the thread (PYTHONIOENCODING names no handler). + for stream in (sys.stdout, sys.stderr): + try: + stream.reconfigure(errors="backslashreplace") + except Exception: + pass + _take_std_fds() + + +def _take_std_fds(): + global _STD_IN, _STD_OUT, _STD_ERR + _STD_IN, _STD_OUT, _STD_ERR = os.dup(0), os.dup(1), os.dup(2) + _own(_STD_IN, "cannot give scripts their stdin any more", "the runtime's copy of fd 0") + _own(_STD_OUT, "cannot reach the output pipes any more", "the runtime's copy of fd 1") + _own(_STD_ERR, "cannot reach the output pipes any more", "the runtime's copy of fd 2") + _enable_faulthandler(_STD_ERR) + + +def _enable_faulthandler(fd): + # A crash's own text goes where the host reads it, a pipe that outlives the + # process: faulthandler adds a traceback for a segfault or an abort, where + # CPython alone writes nothing. Python 3.14 adds the C stack after it, + # native frames a script can't act on, which would push the error's own + # line out of the end of it that the host logs, so it's left out. + faulthandler.enable(file=fd, all_threads=False, + **({"c_stack": False} if sys.version_info >= (3, 14) else {})) + + +def _identity(fd): + # Which file the fd is: its type, device and inode. Not its whole mode: + # macOS's fstat sets a socket's permission bits from its state, so they + # change when its write side is shut down or its peer ends it, as when a + # host ends fd 3 after the exit op, while the fd is still the runtime's own. + st = os.fstat(fd) + return (stat.S_IFMT(st.st_mode), st.st_dev, st.st_ino) + +def _own(fd, stops, which): + _OWN_FDS[fd] = (stops, which, _identity(fd)) -class CappedIO(io.StringIO): - """StringIO that stops recording past a cap (keeps a truncation flag).""" - def __init__(self, cap): +def _lost(fd): + # The runtime's line for one of its own fds that it can't use any more, in + # the words the host's advice reads the fd from (fdTakenFromRuntime in + # py-runner.ts): a script closed or replaced it, or shut down the socket + # under it, which only a failed write shows. + stops, which, _ = _OWN_FDS[fd] + return stops + ": a script closed, shut down or replaced " + which + + +def _check_own_fd(fd): + # A script closed or replaced one of the runtime's own fds: say which and + # end, before the runtime writes to a file that isn't its own any more. + own = _OWN_FDS.get(fd) + if own is None: + return + try: + same = _identity(fd) == own[2] + except OSError: + same = False + if not same: + _fatal(_lost(fd)) + + +def _check_own_fds(): + for fd in list(_OWN_FDS): + _check_own_fd(fd) + + +def _diag(text): + # A note of the runtime's own for the host's log, through fd 2's copy or, + # if a script closed or replaced that, fd 2 itself, and only while it's + # one of the two outputs the host gave the runtime, as the runtime found + # them at its start: a note never goes into a file a script put in its + # place. A background script's journal is one stream, stdout's pipe, so + # there the note goes through fd 1's copy first, after the lines the + # script wrote before it. It waits for room on a pipe a command left non-blocking, + # as the marks do (_write_some). + data = ("[pytc-runtime] " + text + "\n").encode("utf-8", "backslashreplace") + pipes = [_OWN_FDS[fd][2] for fd in (_STD_OUT, _STD_ERR) if fd in _OWN_FDS] + for fd in ((_STD_OUT, 2, _STD_ERR) if BACKGROUND else (_STD_ERR, 2)): + try: + if fd is not None and _identity(fd) in pipes: + _write_all(fd, data) + return + except OSError: + pass + + +def _write_some(fd, data): + # A command can leave the output pipes non-blocking: they're one open file + # with the runtime's fds 1 and 2 and its copies. A full pipe then waits for + # room, where a bare write would raise BlockingIOError. + while True: + try: + return os.write(fd, data) + except BlockingIOError: + select.select([], [fd], []) + + +def _write_all(fd, data): + while data: + data = data[_write_some(fd, data):] + + +class _FdWriter(io.RawIOBase): + # A script's fd 1 or 2 by number (_write_some), which closing never closes. + def __init__(self, fd): super().__init__() - self._cap = cap - self.truncated = False - - def write(self, s): - if not isinstance(s, str): - s = str(s) - if self.tell() >= self._cap: - self.truncated = True - return len(s) - remaining = self._cap - self.tell() - if len(s) > remaining: - self.truncated = True - return super().write(s[:remaining]) - return super().write(s) - - -class LogTee: - """Background-mode writer: line-buffered file append + rolling tail. - - The file is the agent-readable journal (their read/grep/shell tools work - on it); the tail rides the exec_result so the host can build a crash - envelope without re-reading the file. - """ + self._fd = fd + # Set while _drop_held empties the writer above it. + self.dropping = False + + def writable(self): + return True + + def fileno(self): + return self._fd + + def write(self, data): + if self.dropping: + return len(data) + return _write_some(self._fd, data) + + +def _script_writer(fd): + # Line-buffered, as Python's own stdout is on a terminal: each line reaches + # the host as it's printed, in order with what the commands a script runs + # write, for one write per line. A partial line waits for its newline, a + # flush or the end of the exec. It writes to the fd by number, so one an + # earlier script kept, such as a logging handler's, writes into the exec + # that's running. + writer = io.TextIOWrapper(io.BufferedWriter(_FdWriter(fd)), encoding="utf-8", + errors="backslashreplace", newline="\n", line_buffering=True) + _script_writers.add(writer) + return writer + + +def _open_streams(): + # fds 0, 1 and 2 as the host set them, whatever the last script did. + os.dup2(_STD_IN, 0) + os.dup2(_STD_OUT, 1) + # A background script's journal is one stream: its stderr joins its stdout's + # pipe, so their lines keep the order they were written in. + os.dup2(_STD_OUT if BACKGROUND else _STD_ERR, 2) + # And blocking, as the host set them: O_NONBLOCK belongs to a pipe's open + # file, which dup2 doesn't touch, so one a command set would reach every + # later script and command. + os.set_blocking(_STD_OUT, True) + os.set_blocking(_STD_ERR, True) + return _script_writer(1), _script_writer(2) + + +def _off_channel(): + # A script can point its fd 1 or 2 at the runtime's own channel, as with + # os.dup2(4, 1). What its writers still hold would then go into the channel + # ahead of the script's result, which the host couldn't read, and the exec + # would wait out its time limit. So at the script's end, before anything is + # flushed, one that is fd 3 or fd 4 goes back to its pipe, as each exec's + # start puts it (_open_streams). What a script or its commands write onto + # the channel, or read from it, can still garble the runtime's messages: the + # script's own while it runs, and, from a command it started holding the + # channel, a later script's for as long as that command runs. A script + # whose exec_result is garbled waits out its time limit and the grace, and + # the interpreter is restarted. Only a channel per script would end that. + # A copy a script closed or replaced is left alone, so nothing goes into + # a file of the script's: the check at the exec's end ends the runtime. + channel = [_OWN_FDS[fd][2] for fd in (3, 4) if fd in _OWN_FDS] + for fd, pipe in ((1, _STD_OUT), (2, _STD_OUT if BACKGROUND else _STD_ERR)): + try: + if _identity(fd) in channel and _identity(pipe) == _OWN_FDS[pipe][2]: + os.dup2(pipe, fd) + except OSError: + pass + + +def _note(stream, text): + # The runtime's own line in a script's stream, which may have broken under + # it, or whose write the script may have replaced with its own, which may + # raise anything, SystemExit included: the line then goes through the + # runtime's own note (_diag), and the exec's end goes on. + try: + stream.write(text) + except BaseException: + _diag(text.strip("\n")) + + +def _flush_writers(): + # Send what the script left in its writers, the ones it kept from earlier + # execs too, at the exec's end and before a fork. A script can put its own + # code in one, as by replacing its flush, which may raise anything, + # SystemExit included: what it raises is said, but for the ValueError + # _flush takes as a closed writer, and the exec's end or the fork goes on, + # as for a script's own object (_flush_script_streams). + for writer in list(_script_writers): + try: + _flush(writer) + except BaseException as exc: + _diag("some of what the script wrote couldn't be sent (" + _said(exc) + ")") + + +def _flush_script_streams(): + # What a script left in a wrapper of its own that it put in sys.stdout or + # sys.stderr goes out in its exec, as Python flushes them at exit. The + # runtime's own writers are _flush_writers', and a wrapper the script + # keeps anywhere else is its own. Compared by identity: the weak set's + # membership test would hash the object, which a script's may refuse. + writers = list(_script_writers) + for stream in (sys.stdout, sys.stderr): + # A script's own object runs its own code here, which may raise + # anything, SystemExit included: it's said, and the exec's end goes on. + # A closed one's ValueError is passed over, as _flush passes over the + # runtime's own: closing it flushed what it held. + try: + if any(stream is w for w in writers) or not callable(getattr(stream, "flush", None)): + continue + _flush(stream) + except BaseException as exc: + _diag("some of what the script wrote couldn't be sent (" + _said(exc) + ")") + + +def _said(exc): + # An exception's text, or its type's name when it has none, as SystemExit() + # hasn't, or when a script's own exception class refuses to give it. None + # of the script's code runs past the try: the text is taken as an exact str, + # since a str subclass's own methods run on truth and concatenation, and the + # name comes through type's own descriptor, as _brief's does, so a + # metaclass's doesn't run, and as an exact str too, since a script can set + # a class's __name__ to a str subclass. + try: + text = str.__str__(str(exc)) + except BaseException: + text = "" + return text or str.__str__(type.__dict__["__name__"].__get__(type(exc))) + + +def _flush(writer): + try: + writer.flush() + except ValueError: + # A script closed or detached it, or the stream under it: nothing more + # goes through it. A script that closed or detached the writer itself + # flushed it then, and saw any failure of that flush; one that closed + # the stream under it dropped what the writer held, which can't be told + # from nothing here. A ValueError from a flush the script put in its + # place is taken the same way. + return + except BaseException as exc: + # Its fd closed or broke under it, or the script's own flush in its place + # raised, so that much can't be sent. Dropped, + # it can't go out later, once the fd is back, in another exec's output, + # where a writer the script kept would flush it: a writer over the + # runtime's raw writer is (_drop_held), and what a file of the script's + # own holds stays its own, as with any file a script keeps. + _drop_held(writer) + _diag("some of what the script wrote couldn't be sent (" + _said(exc) + ")") + + +def _drop_held(writer): + # Empty a writer whose flush failed, and leave it usable: one of the + # runtime's, or a script's own wrapper over the runtime's raw writer, + # through the runtime's buffer, as TextIOWrapper(sys.stdout.detach()) + # makes, or through a buffer of its own. Emptying one writer leaves + # another wrapper's own buffer as it is; one that shares this writer's + # buffer, as TextIOWrapper(sys.stdout.buffer) does, loses with it what it + # put there, under the same note. + raw = getattr(getattr(writer, "buffer", None), "raw", None) + if not isinstance(raw, _FdWriter): + return + # Through the type's own flush, since a script may have put a flush of + # its own on the writer, which is what failed. + raw.dropping = True + try: + type(writer).flush(writer) + finally: + raw.dropping = False + + +def _before_fork(): + _flush_script_streams() + _flush_writers() + + +if hasattr(os, "register_at_fork"): + # A fork copies what the script's writers hold, and what its own wrappers + # in sys.stdout and sys.stderr hold. Sent first, it reaches the host once, + # and a forked child's writers hold only what the child writes. A wrapper + # the script keeps anywhere else is its own, at a fork as at its end, as + # any buffered file a Python program forks with is. + os.register_at_fork(before=_before_fork) + + +def _write_marks(mark): + # End this exec's share of each stream with its mark: everything the + # script wrote, and all of every command it waited for, is ahead of it. + # A mark that can't be written, on a fd the check before it passed, as + # after a script shut down the socket under fd 1 or 2, ends the + # interpreter, which settles the exec rather than leave it waiting out + # its time limit for the mark. + # The stream the runtime's notes go to gets its mark last, so the line + # _fatal writes when the other's can't be written is still in this exec's + # share: stderr's, or a background script's journal, stdout's (_diag). + data = mark.encode("utf-8") + for fd in ((_STD_ERR, _STD_OUT) if BACKGROUND else (_STD_OUT, _STD_ERR)): + try: + _write_all(fd, data) + except OSError as exc: + _fatal(_lost(fd) + " (" + str(exc) + ")") - def __init__(self, path, tail_chars): - self._fh = None - self._written = 0 - self._tail = "" - self._tail_chars = tail_chars - self.truncated = False - if path: - try: - os.makedirs(os.path.dirname(path), exist_ok=True) - self._fh = open(path, "a", encoding="utf-8", errors="replace") - except Exception as exc: - REAL_STDERR.write("[pytc-runtime] cannot open log file: " + str(exc) + "\n") - REAL_STDERR.flush() - - def write(self, s): - if not isinstance(s, str): - s = str(s) - self._tail = (self._tail + s)[-self._tail_chars:] - if self._fh is not None and not self.truncated: - if self._written + len(s) > LOG_FILE_CAP_BYTES: - self.truncated = True - try: - self._fh.write("\n[log capped at " + str(LOG_FILE_CAP_BYTES) + " bytes]\n") - self._fh.flush() - except Exception: - pass - else: - try: - self._fh.write(s) - if "\n" in s: - self._fh.flush() - self._written += len(s) - except Exception: - pass - return len(s) - - def flush(self): - if self._fh is not None: - try: - self._fh.flush() - except Exception: - pass - - def close(self): - if self._fh is not None: - try: - self._fh.close() - except Exception: - pass - self._fh = None - - def tail(self): - return self._tail + +def _fatal(why): + # The runtime can't go on, as when a script closed one of the runtime's own + # fds under it: say why through its copy of fd 2, which the host reads + # unless a script replaced that too, or shut down the socket under it + # (_diag), and end, so the host settles + # the exec on the exit with the output that arrived, rather than waiting + # out its time limit for a result that can't come. + _diag(why + "; exiting") + os._exit(70) # Persistent interpreter state (mirrors container reuse: variables survive @@ -228,6 +582,9 @@ _current_exec_task = None # 13. a previous script installed an eager task factory -> no effect: the # runtime builds its script task directly, and the state is in place # before the script's first step +# 14. the flush before a fork the script makes -> queued, as in 4; the fork +# goes on, and the cancellation lands at the script's next await, or the +# next signal stops a script that blocks, as in 6 _PENDING = "pending" _REQUESTED = "requested" _DELIVERED = "delivered" @@ -413,15 +770,14 @@ async def wake_agent(payload=None): def handle_init(msg): - global CALL_TIMEOUT_S, BACKGROUND, LOG_PATH, TAIL_CHARS + global CALL_TIMEOUT_S, BACKGROUND timeout = msg.get("call_timeout_s") if isinstance(timeout, (int, float)) and timeout > 0: CALL_TIMEOUT_S = float(timeout) BACKGROUND = bool(msg.get("background")) - LOG_PATH = msg.get("log_path") or None - tail = msg.get("tail_chars") - if isinstance(tail, int) and tail > 0: - TAIL_CHARS = tail + # A background script's stderr joins its stdout's pipe (_open_streams), so + # a crash's traceback goes there too, after the lines the script wrote first. + _enable_faulthandler(_STD_OUT if BACKGROUND else _STD_ERR) # Remove functions injected by a previous init that are no longer present # (tool list changed) without touching user-defined globals. @@ -451,17 +807,34 @@ def handle_init(msg): SCRIPT_GLOBALS.pop("wake_agent", None) -async def _run_script(exec_id, code, st): - global _current_exec_id, _current_exec_task, _stop - if BACKGROUND: - out = LogTee(LOG_PATH, TAIL_CHARS) - err = out # interleave, terminal-style; tail is shared +def _leave_forked(code, err): + # A process the script forked ends on sys.exit() as Python ends: with its + # code, 0 for none, or 1 after writing a code that isn't a number, and with + # what it wrote sent. The atexit handlers are the parent's, so it skips + # them, as multiprocessing's children do. + if code is None: + status = 0 + elif isinstance(code, int): + status = code & 0xFF else: - out = CappedIO(OUTPUT_CAP_CHARS) - err = CappedIO(OUTPUT_CAP_CHARS) + status = 1 + try: + text = str(code) + "\n" + except BaseException: + text = "" + _note(err, text) + _flush_script_streams() + _flush_writers() + os._exit(status) + + +async def _run_script(exec_id, code, mark, st): + global _current_exec_id, _current_exec_task, _stop + _check_own_fds() + out, err = _open_streams() old_out, old_err, old_in = sys.stdout, sys.stderr, sys.stdin sys.stdout, sys.stderr = out, err - sys.stdin = io.StringIO("") # input() must not steal the protocol channel + sys.stdin = io.StringIO("") # input() ends at once, as fd 0's /dev/null does for a command return_code = 0 try: compiled = compile(code, "