fix(code-execution)!: the runtime's channel moves to fds 3 and 4, and the host reads a script's stdout and stderr

changelog.d/fix-code-execution-output-to-host.breaking.md

@@ -0,0 +1 @@
+- **Hosts:** the `code_execution` interpreter now takes its channel to the host on its file descriptors 3 and 4, and `PYTHON_RUNTIME_SOURCE`, which the package exports, speaks a new protocol there. **Who needs to act:** a host whose `pythonPath` starts Python through a wrapper that doesn't pass fds 3 and 4 on, such as one that runs it in a container, under sudo with its default settings, or from a Python script through `subprocess`, whose `close_fds` is on by default, and a host that runs `PYTHON_RUNTIME_SOURCE` itself rather than through `PyRunner`. Until it acts, every script fails to start, and the result says the host's channel didn't reach the interpreter: the runtime says so on stderr and exits with 78 (EX_CONFIG). **Migration:** have the wrapper `exec` Python, or pass fds 3 and 4 on. A host that runs the runtime itself opens fd 3 for its messages to the runtime and fd 4 for the runtime's, gives it `/dev/null` as fd 0, reads its fds 1 and 2 itself, puts a `mark` in each `exec`, and takes the script's output on each stream up to that mark, which the runtime writes once the script returns: `exec_result` now carries only the return code. It stops the runtime with the `exit` op, since closing fd 3 while a script runs now ends the runtime with exit code 70 and a line saying a script closed, shut down or replaced fd 3. `init` no longer takes `log_path` or `tail_chars`, so the runtime writes no journal for a background script and keeps no tail: the host writes both from fds 1 and 2, where a background script's stderr joins its stdout. **Unchanged by this change:** `PyRunner`'s interface, `ExecResult` included, and a `pythonPath` that names an interpreter or a wrapper that `exec`s one, of which it needs nothing more. The interpreter's new working directory, where a wrapper's own relative paths now resolve, comes from #193, whose entry says what it changes. No sibling package needs a new version.

changelog.d/fix-code-execution-output-to-host.fixed.md

@@ -0,0 +1,8 @@
+- What the commands a `code_execution` script runs write to stdout and stderr now comes back in the script's result, beside the script's own prints, line by line in the order written on each stream: `subprocess` without `capture_output`, `os.system`, and C code writing to file descriptors 1 and 2. It used to reach only the host's log, and a line a command wrote without a newline ran into the runtime's own reply, so the script waited out its time limit. A command with a lot of output now fills the result as the script's own prints do: each stream keeps up to 2,000,000 characters, counted as UTF-16 code units, so a character outside the Basic Multilingual Plane, such as an emoji, counts as two, and a long result spills to a workspace file, as any result does. Output belongs to the script running when it arrives: a command a script left running writes into the result of a later script running then, and into the host's log between scripts or once its interpreter has been replaced. The tool's description now says that what the script and the commands it runs print comes back, and that a background script's output streams to a log file when the agent has a writable workspace. That changes the tool definitions every request carries, and by [Anthropic's prompt-caching documentation](https://platform.claude.com/docs/en/build-with-claude/prompt-caching#what-invalidates-the-cache), a prompt cache written before the upgrade isn't reused.
+- A command a `code_execution` script runs now reads the end of its input at once: its standard input is `/dev/null`. It used to read the runtime's channel from the host, so a command that reads its input, such as `cat`, or `ssh` without `-n`, took the host's messages, and the script failed or waited out its time limit.
+- A script that closes or redirects its file descriptors 0, 1 or 2 leaves the next script its output: the runtime's channel to the host has file descriptors of its own, 3 and 4, which no command a script runs is given, and each script starts with 0, 1 and 2 as the host set them, blocking even after a command set one non-blocking. A script that points 1 or 2 at the runtime's channel gets it put back at its end, so a partial line it left there reaches its result; what a script or its commands write onto that channel, or read from it, can still garble the runtime's messages: the script's own, and, from a command started holding the channel, a later script's for as long as that command runs. A script whose result is garbled that way waits out its time limit and the grace, and the interpreter restarts, so the next script's variables are gone. A script's `os.close(1)` used to leave it waiting out its time limit, and its `os.close(0)` sometimes left the next script doing so. A script that closes or replaces one of the runtime's own file descriptors, 3, 4, or its copies of 0 to 2, now ends the interpreter by the end of that script at the latest, and the result says which and what to check, as does one that shuts down the socket under file descriptor 1, 3 or 4. That line is lost when the script also closed or replaced both the runtime's copy of file descriptor 2 and file descriptor 2 itself, or shut down the socket under them, and the result leaves it out once the script's stderr has passed its 2,000,000-character cap.
+- What a script and its commands wrote before it was stopped, as far as it had reached the host, now comes back ahead of the message saying why, when it's killed at its time limit, aborted, or its interpreter crashes; it used to be lost. A partial line still in the script's writer when the interpreter is killed or crashes is lost with it. A crash's own text comes back too, and goes to the host's log: a fatal error's, which goes to the script's file descriptor 2 wherever the script pointed it, and now a traceback for a segfault or an abort, which faulthandler writes where CPython alone wrote nothing.
+- A background script's journal now holds its commands' output too, each line as it arrives, and ends saying why the host stopped the script, when it did, a time limit's note included, unless the journal reached its 50 MB cap, now counted in bytes, as its cap line says, or a write to it failed. A background script that crashes leaves the last of what it wrote, its crash's text included and after it, in the message that wakes its agent, where that held only the message about the exit. A journal that can't be opened or written says so once in the host's log. A background script started where no workspace is writable is told that `wake_agent()` reaches its agent and, if the script dies, the last of its output reaches it too, where it was told its output wasn't retrievable.
+- A script that prints a lone surrogate, such as `"\udcff"`, now gets its output back with it written as `\udcff`, rather than as U+FFFD.
+- A script's prints now reach the host as each line is complete, so a partial line waits for its newline, a flush or the end of the script, as Python's stdout does on a terminal, and what a wrapper of the script's own in `sys.stdout` or `sys.stderr` holds is flushed at the script's end, as Python's exit would. Printing costs more than it did: 100,000 short `print` calls took about 0.36 s, against about 0.15 s before, on Linux with Node 24.22.0 and Python 3.13.5.
+- An interpreter that exits as it starts now says in the result what it wrote to stderr, up to its last 2,000 characters, with "…" where they're cut, and one ended by a signal as it starts names the signal. Output that arrives while its interpreter runs no script, such as from a command a script left running, even once that interpreter has ended, reaches the host's log as `stdout while its interpreter runs no script: …` or `stderr while its interpreter runs no script: …`, each piece cut at 2,000 characters, where stdout's used to read `non-protocol stdout line: …`: up to 65,536 characters of such lines, counted as UTF-16 code units, on each pipe from the start of its interpreter's last script, then a line saying the host drops the rest while the interpreter runs no script. While the interpreter lives, the host reads such output as it's written; once the host has torn the interpreter down, as at an idle reclaim, it reads it at 1 MiB a second at most, so a command left running that writes without end slows to that rather than costing the host a core, and keeps running. A host that turns the idle reclaim off reads such a command at full speed for as long as its interpreter lives. When the host's messages stop reaching an interpreter, the host now gives it a moment to exit on its own before killing it, so one that died, as by running out of memory, reads as its own exit rather than as one that stopped reading; and the stopped-reading message no longer says a script closed its standard input: it says the interpreter stopped reading the host's messages, and to check whether the script closes, shuts down or replaces file descriptor 3.