From e362caaa6a870b45f254fe5b07e8719242826c2f Mon Sep 17 00:00:00 2001 From: Simon Qi Date: Thu, 27 Aug 2026 10:14:15 +1200 Subject: [PATCH 1/4] Skip the Codex update prompt by starting the TUI with an initial prompt (#110) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `~/.codex/version.json` learns of a release, every hand-started `codex` stops at `1. Update now / 2. Skip / 3. Skip until next version` **before the Session registers** — no thread on the daemon, no rollout, indefinitely. One probe that started two seconds before the check landed booted normally; the three after it did not. The run that followed was green because Simon had updated the CLI by hand, and the gate returns at 0.151.0. The open question it left was whether the harness may arrange that gate the way it arranges `TrustGate`, by writing `dismissed_version` into a file the user owns. It may not, and it does not have to. A non-empty `PROMPT` argument makes the TUI skip the prompt outright — `codex [OPTIONS] [PROMPT]`, decided at `tui/src/lib.rs:995-996` and byte-identical at 0.149.1 and 0.150.0 (#107 Q6): let skip_update_prompt = cli.prompt.as_ref().is_some_and(|p| !p.is_empty()); So the Codex lane launches with the words its first turn would have typed anyway. Nothing of the user's is edited, `version.json` is not written, and there is nothing for the run to revoke afterwards — the gate is *skipped*, not arranged, which is the one place this design takes the opposite decision to `TrustGate`. The emptiness test is the mechanism, which is why the argument is a real instruction rather than a placeholder that could quietly shrink to `""`. **The one route not taken, and why it is written down as a trap.** `-c check_for_update_on_startup=false` suppresses the prompt *and* makes the TUI refuse the shared daemon — `can_reuse_implicit_local_daemon` requires `cli_kv_overrides.is_empty()` (`tui/src/lib.rs:919-921`) — so it would silently destroy the attachment this run exists to measure. A harness that reaches for `-c` to solve a boot gate is measuring nothing. **It is an extra turn, not a replacement, and two steps depend on that.** `stable name` requires a name held across a Stop it drives itself, and `stop notice` marks the chat immediately before that turn; a Stop crossed at launch would predate the mark and its notice would be unfindable. So `stable name` still types `ACKNOWLEDGE`, and the boot turn asks the Session nothing the run does not already ask. It uses no tools, so it cannot raise a permission before `approval` is there to answer one, and it changes nothing `codex_turn_policy` reads: the sandbox is still the lane's `workspace-write` pin, and the `turn_context` that step grades is `relay`'s, the last one written. Two consequences recorded rather than left to be discovered. Codex writes its rollout when the first *turn* starts, so the rollout now exists before `roster` runs and this lane's ground truth carries a real `session_id` at the first read instead of `""`; the pid join underneath is unchanged. And at 0.150.0 the first user message spawns an ephemeral title-generation thread (#107, recommendation 2), so that phantom row — if it is one — appears during boot rather than during `stable name`; `roster` asserts only that *this* Session has a row, so an extra row does not fail it, and counting the rows stays #79's observation item, one step earlier than it was. What the verification run can prove is bounded, and the bound is stated in the design rather than discovered later: on 2026-08-27 this machine's `version.json` read `latest_version 0.150.0` against a 0.150.0 CLI, so `get_upgrade_version` returns `None` and the prompt is **not armed**. The run proves that a TUI started this way registers on the daemon and that `version.json` is not written by us; the skip itself stands on the source at both tags and will be observed for real on the first run after 0.151.0 ships. Legacy: no citation, checked and empty. Gen 1's Codex path forwarded the user's own argv verbatim (`legacy@1d32845:bridge/codex.py:1971-1975`) and nowhere reads or writes the update cache — `check_for_update`, `version.json` and the prompt's own words appear nowhere in that tree. It had no automated real-environment acceptance to hit the gate from, so there is no prior implementation to port (ADR 0010's rule, answered in the negative). Harness only; no product code, and no version pin. 1496 tests pass; ruff clean. Co-Authored-By: Claude Opus 5 (1M context) --- tests/acceptance/hand_started.py | 29 +++++++++++++++++++++++++++++ tests/acceptance/journey.py | 23 ++++++++++++++++++++++- 2 files changed, 51 insertions(+), 1 deletion(-) diff --git a/tests/acceptance/hand_started.py b/tests/acceptance/hand_started.py index 43cdf8a..8ba2c26 100644 --- a/tests/acceptance/hand_started.py +++ b/tests/acceptance/hand_started.py @@ -33,6 +33,35 @@ written to the run's artifacts as evidence a human can read; every assertion in `journey.py` rests on the roster, the rollout, the filesystem, `engine.log` or the chat. + +A fourth, measured on #110 (2026-08-27, `codex-cli` 0.150.0): + +* **The Codex lane launches with an initial prompt, and that argument is the + whole update gate.** From the instant `~/.codex/version.json` learns of a + release, every hand-started `codex` stops at `1. Update now / 2. Skip / 3. Skip + until next version` **before the Session registers** — no thread on the daemon, + no rollout, indefinitely; #105 watched one probe boot normally and the three + that started after the check landed die there, 90 seconds apart. Codex skips + that prompt outright when it is handed a non-empty `PROMPT` argument + (`codex [OPTIONS] [PROMPT]`), decided at `tui/src/lib.rs:995-996` and + byte-identical at 0.149.1 and 0.150.0 (read on #107): + + let skip_update_prompt = cli.prompt.as_ref().is_some_and(|p| !p.is_empty()); + + So `journey.CODEX` passes the words its first turn would have typed anyway. + **The emptiness test is the mechanism** — an empty string is not a skip — which + is why the argument is the journey's own instruction and not a placeholder that + could quietly shrink to `""`. + + The alternative, `-c check_for_update_on_startup=false`, suppresses the prompt + *and* makes the TUI refuse the shared daemon (`can_reuse_implicit_local_daemon` + requires `cli_kv_overrides.is_empty()`, `tui/src/lib.rs:919-921`): it would + silently destroy the attachment the run exists to measure. **Never reach for + `-c` to solve a boot gate.** + + This gate is *skipped*, not arranged, and that is what makes it unlike + `TrustGate`: nothing of the user's is edited, `version.json` is not written, and + there is nothing for the run to revoke afterwards. """ from __future__ import annotations diff --git a/tests/acceptance/journey.py b/tests/acceptance/journey.py index f72e64c..e807b7e 100644 --- a/tests/acceptance/journey.py +++ b/tests/acceptance/journey.py @@ -301,11 +301,32 @@ class Lane: #: a trusted workspace already gives (measured 2026-08-26 on the failing run's #: own rollout, `turn_context.sandbox_policy = workspace-write`), so the flag #: fixes the ground rather than moving it. +#: +#: The last argument is the **initial prompt**, and it is here to get past the +#: update gate rather than to ask for anything (#110; the measurement is in +#: `hand_started`'s module docstring). `codex [OPTIONS] [PROMPT]` — the prompt is +#: positional, so it goes after the flags and stays last. Two things follow, and +#: both are stated because a reader will otherwise meet them as surprises: +#: +#: * **It is an extra turn, not a replacement.** `stable name` still types +#: `ACKNOWLEDGE` itself, because it requires a name held across a Stop it +#: drives, and `stop notice` marks the chat immediately before that turn — a +#: Stop crossed at launch would predate the mark and the notice would be +#: unfindable. The words are `ACKNOWLEDGE`'s so the boot turn asks the Session +#: nothing the run does not already ask, and it uses no tools, so it cannot +#: raise a permission before `approval` is there to answer it, and it leaves +#: `codex_turn_policy` reading the same ground: the sandbox is this lane's pin +#: and the `turn_context` that step grades is `relay`'s, the last one written. +#: * **The rollout now exists before `roster` runs.** Codex writes its rollout +#: when the first *turn* starts, so this lane's `ground_truth` carries a real +#: `session_id` at the first read instead of the `""` it used to carry — the +#: evidence line `roster` prints changes shape, and the pid join it rests on +#: does not. CODEX = Lane( name="codex", agent="codex", binary="codex", - arguments=("--sandbox", "workspace-write"), + arguments=("--sandbox", "workspace-write", ACKNOWLEDGE.words), # Measured 2026-08-27 through the shared daemon with the product's own pin # and no sandbox override, on codex-cli 0.149.1 and again on 0.150.0 over a # 0.149.1 app-server: a write to a path outside the workspace raises From 70376418f920976219ae0727206196632d3e301a Mon Sep 17 00:00:00 2001 From: Simon Qi Date: Thu, 27 Aug 2026 10:26:59 +1200 Subject: [PATCH 2/4] Wait out the turn the launch starts, before anything is typed (#110 review) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Both axes of the Codex review of the change before it found the same P1, and it was a false green rather than a red. A prompt on the command line starts a turn **asynchronously**: `session.start()` returns at once and nothing downstream waited for it. `stable name` then takes `stop notice`'s chat mark and submits the walk's first turn — so a boot turn still running at that moment puts its own Stop Notice on the far side of the mark, and `stop notice` passes on the notice for a turn nobody drove. The two turns carry the same words, so no reader of the chat could tell them apart either. Neither reviewer had to run it to see it, and neither would a Session that answered slowly. `Walk.settle_boot_turn` waits the boot turn out before a word is typed or a mark is taken, by the same rule `_drive_turn` uses — the agent's own record grew and then stopped growing, extracted as `_turn_ended` so a boot turn cannot be judged by a looser rule than the turns it runs ahead of. It is arranged, not graded (`TrustGate`'s and `arm_switches`' class): it lands in `verdict.observations` as `boot turn` with its seconds, and a lane still unsettled after twice the turn figure is **blocked** rather than typed into. Twice, because the wait covers boot *and* one turn, and boot alone has been measured at the whole figure on this machine — a `codex` sat in `starting MCP servers` for a full 180s ground-truth wait on 2026-08-26. The second finding was the string: the standards axis read `arguments=("--sandbox", "workspace-write", ACKNOWLEDGE.words)` as three interchangeable words hiding three rules — must be last, must be non-empty, starts a turn. So the boot prompt is now its own field on the lane (`Lane.boot_prompt`), the argv is composed in one place that holds the two command-line rules (`hand_started.launch_arguments`, which refuses an empty prompt because emptiness is exactly what the update gate tests), and the third rule is `settle_boot_turn`'s whole reason to exist. The Claude lane passes `None` and is launched silent, as it was. Exercised against fakes before committing: a boot turn that grows and settles is measured and observed, one that never stops growing blocks the lane with the record size and the screen tail, and the Claude lane does nothing at all. 1496 tests pass; ruff clean. Co-Authored-By: Claude Opus 5 (1M context) --- tests/acceptance/conftest.py | 2 +- tests/acceptance/hand_started.py | 31 ++++++++- tests/acceptance/journey.py | 112 +++++++++++++++++++++++++++---- 3 files changed, 131 insertions(+), 14 deletions(-) diff --git a/tests/acceptance/conftest.py b/tests/acceptance/conftest.py index 9e7b4bc..fab484d 100644 --- a/tests/acceptance/conftest.py +++ b/tests/acceptance/conftest.py @@ -398,7 +398,7 @@ def hand_started_session( # noqa: ANN201 session = hand_started.HandStartedSession( lane=lane.name, binary=binary, - arguments=lane.arguments, + arguments=hand_started.launch_arguments(lane.arguments, lane.boot_prompt), workspace=config.workspace, environment=terminal_environment, journal=journal, diff --git a/tests/acceptance/hand_started.py b/tests/acceptance/hand_started.py index 8ba2c26..ea2ed31 100644 --- a/tests/acceptance/hand_started.py +++ b/tests/acceptance/hand_started.py @@ -61,7 +61,10 @@ This gate is *skipped*, not arranged, and that is what makes it unlike `TrustGate`: nothing of the user's is edited, `version.json` is not written, and - there is nothing for the run to revoke afterwards. + there is nothing for the run to revoke afterwards. What it *is* unlike an + ordinary flag in is that **a prompt on the command line is a turn**, running + before the walk has asked for anything — so `journey.Walk.settle_boot_turn` + waits it out before a word is typed or a chat mark is taken. """ from __future__ import annotations @@ -143,6 +146,32 @@ def resolve(binary: str, path_value: str) -> Path | None: return Path(found) if found else None +def launch_arguments(flags: tuple[str, ...], boot_prompt: str | None) -> tuple[str, ...]: + """A lane's argv: its flags, then its boot prompt — last, and never empty. + + The two rules this holds are both load-bearing and neither is visible in a + tuple literal. **Last**, because both agents take the prompt as a positional + (`codex [OPTIONS] [PROMPT]`), so a prompt written before a flag is read as + that flag's value. **Never empty**, because emptiness is what the update gate + tests (see the fourth measurement above): `""` is a prompt that starts no + turn and skips no gate, and the TUI would stop at the update prompt exactly + as if nothing had been passed — silently, and only in the weeks after a + release, which is the worst way for a harness to be wrong. + + A lane with no boot prompt passes `None` and gets its flags back. That is the + Claude lane, and it is the ordinary case: a Session nobody has typed into is + what the walk's first step wants to find. + """ + if boot_prompt is None: + return flags + if not boot_prompt.strip(): + raise ValueError( + "a boot prompt must be non-empty: an empty PROMPT is not a skipped update gate, " + "it is a launch that stops at the gate with nothing to show for it" + ) + return (*flags, boot_prompt) + + class SessionRefused(RuntimeError): """The command would not start, so there is no Session to judge anything on.""" diff --git a/tests/acceptance/journey.py b/tests/acceptance/journey.py index e807b7e..3f90784 100644 --- a/tests/acceptance/journey.py +++ b/tests/acceptance/journey.py @@ -36,6 +36,12 @@ words arriving and the permission that turn raises — because a relayed instruction that needs a permission is exactly the shape the product has to survive, and running it twice would prove less at twice the cost. + +The Codex lane runs a sixth that no step drives: it is *launched* with a prompt, +because that is what carries it past Codex's update gate (#110), and a prompt on +the command line is a turn. No step observes it and `Walk.settle_boot_turn` waits +it out before the walk begins — a turn still running when the first step types is +a turn whose Stop lands where a later step is looking for a different one. """ from __future__ import annotations @@ -78,6 +84,15 @@ #: cadence once there is one. DISCOVERY_SECONDS = 30.0 +#: How long a boot turn gets, as a multiple of the far side's turn figure. +#: Derived rather than guessed: `settle_boot_turn` waits out a TUI's **boot and** +#: its first turn, and boot alone has been measured at the whole turn figure on +#: this machine — a `codex` sat in `starting MCP servers` for a full 180s +#: ground-truth wait on 2026-08-26 (`hand_started.codex_ground_truth`). So two of +#: them, one for each half, and a lane still unsettled after that is blocked +#: rather than typed into. +BOOT_TURN_ALLOWANCE = 2.0 + #: What the escalated permission announcement says (`core/approvals.py:43-46`). #: Matched rather than quoted whole: the tool name and the detail are the agent's, #: and this run does not get to predict them. @@ -214,6 +229,12 @@ class Lane: binary: str #: Arguments the *person* would not normally type, and why each is here. arguments: tuple[str, ...] + #: Words the lane's TUI is **launched with**, or None when it is launched + #: silent. Not one of `arguments`, because it is not a flag: it starts a turn + #: nobody drove, before the walk has asked for anything. `hand_started. + #: launch_arguments` puts it last on the command line and refuses an empty + #: one; `Walk.settle_boot_turn` waits it out before a word is typed. + boot_prompt: str | None #: The words that make this lane's agent spawn a Child Process. "subagent" #: and "sub-agent" appear here on purpose: this string is spoken *to* the #: agent, where it is the agent's own mechanism word and the thing that makes @@ -267,6 +288,10 @@ class Lane: agent="claude", binary="claude", arguments=("--permission-mode", "default"), + # Launched silent. No boot gate of the Codex kind has been measured here — + # `claude` boots into an empty composer — and a Session nobody has typed into + # is what `roster` and `stable name`'s three reads want to find. + boot_prompt=None, relayed=lambda workspace: writing(RELAY_FILE, RELAY_WORD), # The flag the harness passes *is* the whole policy on this lane, and Claude # publishes no per-turn readback of it, so there is nothing to read back and @@ -302,11 +327,10 @@ class Lane: #: own rollout, `turn_context.sandbox_policy = workspace-write`), so the flag #: fixes the ground rather than moving it. #: -#: The last argument is the **initial prompt**, and it is here to get past the -#: update gate rather than to ask for anything (#110; the measurement is in -#: `hand_started`'s module docstring). `codex [OPTIONS] [PROMPT]` — the prompt is -#: positional, so it goes after the flags and stays last. Two things follow, and -#: both are stated because a reader will otherwise meet them as surprises: +#: The **boot prompt** is not a flag and is not here to ask for anything: it is +#: what gets this lane past the update gate (#110; the measurement is in +#: `hand_started`'s module docstring). Three things follow, and each is stated +#: because a reader will otherwise meet it as a surprise: #: #: * **It is an extra turn, not a replacement.** `stable name` still types #: `ACKNOWLEDGE` itself, because it requires a name held across a Stop it @@ -317,6 +341,10 @@ class Lane: #: raise a permission before `approval` is there to answer it, and it leaves #: `codex_turn_policy` reading the same ground: the sandbox is this lane's pin #: and the `turn_context` that step grades is `relay`'s, the last one written. +#: * **The walk waits it out first** (`Walk.settle_boot_turn`). Two turns of the +#: same words are not two turns the harness can tell apart: a boot turn still +#: running when `stop notice`'s mark is taken puts *its* Stop Notice after the +#: mark, and the step would pass on the notice for a turn nobody drove. #: * **The rollout now exists before `roster` runs.** Codex writes its rollout #: when the first *turn* starts, so this lane's `ground_truth` carries a real #: `session_id` at the first read instead of the `""` it used to carry — the @@ -326,7 +354,8 @@ class Lane: name="codex", agent="codex", binary="codex", - arguments=("--sandbox", "workspace-write", ACKNOWLEDGE.words), + arguments=("--sandbox", "workspace-write"), + boot_prompt=ACKNOWLEDGE.words, # Measured 2026-08-27 through the shared daemon with the product's own pin # and no sandbox override, on codex-cli 0.149.1 and again on 0.150.0 over a # 0.149.1 app-server: a write to a path outside the workspace raises @@ -426,6 +455,7 @@ def walk(self) -> None: ) try: self.arm_switches() + self.settle_boot_turn() except LaneBlocked as unarmed: self.journey.skip_rest(str(unarmed)) return @@ -444,6 +474,51 @@ def walk(self) -> None: or "no turn ran", ) + def settle_boot_turn(self) -> None: + """Wait out the turn the *launch* started, before a word is typed at it. + + Not a step, for `arm_switches`' reason: it is how this lane is put in the + state the walk assumes, not a claim about the product. But it is not + optional either, and the failure it prevents is a **false green** rather + than a red. + + A lane with a `boot_prompt` is running a turn from the moment it starts + (#110 — the prompt is what carries it past the update gate). Nothing may + be typed into a Session that is mid-turn, and no chat mark may be taken + while one is in flight: `stable name` drives the walk's first turn and + hands `stop notice` the mark from just before it, so a boot turn that + ends *after* that mark puts its own Stop Notice on the far side of it — + and `stop notice` passes on a notice for a turn nobody drove, having + proved nothing. The two turns carry the same words, so no reader of the + chat could tell them apart afterwards either. + + Over means what it means everywhere else here (`_turn_ended`): the + agent's own record grew and then stopped growing. Nothing is typed and + nothing is asked — this only watches. + """ + if self.lane.boot_prompt is None: + return + started = time.monotonic() + # Resolves the record this waits on. It is the ordinary first call: every + # step reads ground truth through here, and it is cached after the first. + truth = self._ground_truth() + deadline = started + self.far_side.agent_turn_seconds * BOOT_TURN_ALLOWANCE + turn = self._measured("boot prompt", started, self._turn_ended(deadline=deadline)) + if not turn.ended: + raise LaneBlocked( + f"the turn this lane was launched with had not ended after {turn.seconds:.0f}s, " + f"so the walk cannot type into this Session without racing it. The agent " + f"reports {truth.describe()}; its record is {self._record_size()} bytes. " + f"Screen tail: {self.session.screen_tail()[-600:]!r}" + ) + self.journey.observe( + "boot turn", + f"the Session was launched with {self.lane.boot_prompt!r} — the words `stable name` " + f"types anyway — because a non-empty prompt is what skips Codex's update gate " + f"(#110). Waited out before anything was typed: {turn.seconds:.1f}s, record " + f"{self._record_size()} bytes. Arranged by the harness, not judged by it.", + ) + def arm_switches(self) -> None: """Voice off, Message on, Duty on — the text-only mode this whole run exercises. @@ -1033,15 +1108,25 @@ def _drive_turn( """ started = time.monotonic() self.session.submit(instruction.words) + ended = self._turn_ended( + deadline=started + self.far_side.agent_turn_seconds, expect_waiting=expect_waiting + ) + return self._measured(what, started, ended) + + def _turn_ended(self, *, deadline: float, expect_waiting: bool = False) -> bool: + """Whether the turn in flight is over by `deadline`, read from outside. + + Shared by the turns the walk drives and the one the launch starts + (`settle_boot_turn`), because "over" has to mean the same thing for both: + a boot turn judged by a looser rule than the turns it runs ahead of is a + boot turn that can still be running when the next one starts. + """ settled_for = 0.0 last = self._record_size() - ended = False - deadline = started + self.far_side.agent_turn_seconds while time.monotonic() < deadline: time.sleep(3.0) if expect_waiting and self._roster_field("state") == "waiting": - ended = True - break + return True size = self._record_size() if size != last: last, settled_for = size, 0.0 @@ -1050,8 +1135,11 @@ def _drive_turn( # A record that has not grown for two polls after growing at all is a # turn that is over; before it grows at all there is nothing to settle. if size > 0 and settled_for >= 9.0: - ended = True - break + return True + return False + + def _measured(self, what: str, started: float, ended: bool) -> Turn: + """Record one turn on the walk, whoever asked for it.""" turn = Turn(what, time.monotonic() - started, ended) self.turns.append(turn) self.journal("turn", lane=self.lane.name, what=what, seconds=turn.seconds, ended=turn.ended) From 57f5ec980aa64ea2262cb22fd8eacb4a03063b83 Mon Sep 17 00:00:00 2001 From: Simon Qi Date: Thu, 27 Aug 2026 10:34:07 +1200 Subject: [PATCH 3/4] Ask Codex whether the boot turn is over, and wait with every outlet off (#110 re-review) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The re-review retained its P1, and it was right. The wait added in the commit before this one settled for nine seconds of an unchanging record — `_drive_turn`'s rule — and a turn waiting on the model appends nothing either. Silence cannot tell "finished" from "thinking", so the boot turn could still be called over while it was running, and the false green the fix existed to prevent survived it. **Codex says which it is.** Every turn is bracketed in the rollout by `event_msg`/`task_started` and `event_msg`/`task_complete` — measured 2026-08-27 against six real rollouts on this machine (codex-cli 0.150.0): five with balanced counts read *over*, and one live session mid-turn read *in flight*, which is the case the silence rule got wrong. `hand_started.codex_turn_over` counts both and answers only when every turn that started has finished. `_drive_turn` keeps its own rule, unchanged: for a graded turn the ambiguity costs a slow reading, and changing what "over" means for the five turns the walk drives is not this ticket's to do. **And the wait now runs before `arm_switches`, which closes the other half.** Even a perfect end-of-turn reading leaves the product free to publish the boot Stop *after* the mark, because it observes Stops on its own cadence. With the outlets still off for the whole boot turn there is no notice to publish: a fresh engine answers `duty off, message off, voice off` (#60), and Duty coming on reports only what is still actionable (#80) — which an idle Session is not. Both sides are needed and both are now there. The boot prompt and its end-of-turn reading are one value (`hand_started. BootPrompt`), because a lane that gained the prompt without the reading is exactly the defect the review found, and a type that cannot express that is cheaper than a comment asking nobody to. Exercised before committing: `codex_turn_over` against the six real rollouts and against seven synthetic ones — no record, session-but-no-turn, a turn in flight, a turn in flight that has gone quiet, one turn done, a second turn in flight, both done — plus a half-written trailing line, which it tolerates because it is reading a file another process is appending to. `settle_boot_turn` measured and observed a boot turn that completes, blocked the lane on one that does not, and did nothing at all on the Claude lane. 1496 tests pass; ruff clean. Co-Authored-By: Claude Opus 5 (1M context) --- tests/acceptance/conftest.py | 2 +- tests/acceptance/hand_started.py | 67 +++++++++++++++++++-- tests/acceptance/journey.py | 99 +++++++++++++++++--------------- 3 files changed, 118 insertions(+), 50 deletions(-) diff --git a/tests/acceptance/conftest.py b/tests/acceptance/conftest.py index fab484d..90ae771 100644 --- a/tests/acceptance/conftest.py +++ b/tests/acceptance/conftest.py @@ -398,7 +398,7 @@ def hand_started_session( # noqa: ANN201 session = hand_started.HandStartedSession( lane=lane.name, binary=binary, - arguments=hand_started.launch_arguments(lane.arguments, lane.boot_prompt), + arguments=hand_started.launch_arguments(lane.arguments, lane.boot), workspace=config.workspace, environment=terminal_environment, journal=journal, diff --git a/tests/acceptance/hand_started.py b/tests/acceptance/hand_started.py index ea2ed31..ef89ade 100644 --- a/tests/acceptance/hand_started.py +++ b/tests/acceptance/hand_started.py @@ -83,6 +83,7 @@ import threading import time from collections import deque +from collections.abc import Callable from dataclasses import dataclass from pathlib import Path @@ -146,7 +147,65 @@ def resolve(binary: str, path_value: str) -> Path | None: return Path(found) if found else None -def launch_arguments(flags: tuple[str, ...], boot_prompt: str | None) -> tuple[str, ...]: +@dataclass(frozen=True) +class BootPrompt: + """Words a lane's TUI is *launched* with, and how to know that turn is over. + + The two travel together because either alone is a trap. A prompt on the + command line is what carries the Codex lane past the update gate (#110), and + it is also a **turn**, running before the walk has asked for anything; the + reading is how `journey.Walk.settle_boot_turn` knows it may stop waiting. A + lane that gained the first without the second is exactly the false green the + Codex review of #110 found, so the type does not let it. + """ + + words: str + #: Whether the agent's own record shows **no turn in flight**, given where + #: that record is now (or None when there is not one yet). + turn_over: Callable[[Path | None], bool] + + +#: How Codex brackets a turn in its own record. Measured 2026-08-27 against a +#: real rollout on this machine (`rollout-2026-08-27T10-17-38-01a04026…jsonl`, +#: codex-cli 0.150.0): two turns, two `task_started`, two `task_complete`, and +#: the file's last record is the second `task_complete`. +TURN_STARTED = "task_started" +TURN_COMPLETE = "task_complete" + + +def codex_turn_over(rollout: Path | None) -> bool: + """Whether Codex's own record shows no turn in flight: all started, all complete. + + **Not** "the record stopped growing for a while", which is what `_drive_turn` + settles for and what the Codex review of #110 rejected for this use: a turn + waiting on the model appends nothing, so silence is a turn that may be over + and may be thinking. For a graded turn that ambiguity costs a slow reading; + for the boot turn it costs the run its meaning, because a boot turn wrongly + called over is a Stop landing where a later step is looking for a different + one. Codex says which it is, so this asks Codex. + """ + if rollout is None: + return False + started = complete = 0 + try: + with rollout.open() as lines: + for line in lines: + try: + record = json.loads(line) + except json.JSONDecodeError: + continue # a line half-written while this read it + if record.get("type") != "event_msg": + continue + payload = record.get("payload") + kind = payload.get("type") if isinstance(payload, dict) else None + started += kind == TURN_STARTED + complete += kind == TURN_COMPLETE + except OSError: + return False + return started > 0 and complete >= started + + +def launch_arguments(flags: tuple[str, ...], boot: BootPrompt | None) -> tuple[str, ...]: """A lane's argv: its flags, then its boot prompt — last, and never empty. The two rules this holds are both load-bearing and neither is visible in a @@ -162,14 +221,14 @@ def launch_arguments(flags: tuple[str, ...], boot_prompt: str | None) -> tuple[s Claude lane, and it is the ordinary case: a Session nobody has typed into is what the walk's first step wants to find. """ - if boot_prompt is None: + if boot is None: return flags - if not boot_prompt.strip(): + if not boot.words.strip(): raise ValueError( "a boot prompt must be non-empty: an empty PROMPT is not a skipped update gate, " "it is a launch that stops at the gate with nothing to show for it" ) - return (*flags, boot_prompt) + return (*flags, boot.words) class SessionRefused(RuntimeError): diff --git a/tests/acceptance/journey.py b/tests/acceptance/journey.py index 3f90784..b9b9a65 100644 --- a/tests/acceptance/journey.py +++ b/tests/acceptance/journey.py @@ -90,7 +90,8 @@ #: this machine — a `codex` sat in `starting MCP servers` for a full 180s #: ground-truth wait on 2026-08-26 (`hand_started.codex_ground_truth`). So two of #: them, one for each half, and a lane still unsettled after that is blocked -#: rather than typed into. +#: rather than typed into. Blocked, not merely slow: everything after it would be +#: measuring a Session with a turn still running underneath. BOOT_TURN_ALLOWANCE = 2.0 #: What the escalated permission announcement says (`core/approvals.py:43-46`). @@ -229,12 +230,13 @@ class Lane: binary: str #: Arguments the *person* would not normally type, and why each is here. arguments: tuple[str, ...] - #: Words the lane's TUI is **launched with**, or None when it is launched + #: What the lane's TUI is **launched with**, or None when it is launched #: silent. Not one of `arguments`, because it is not a flag: it starts a turn #: nobody drove, before the walk has asked for anything. `hand_started. - #: launch_arguments` puts it last on the command line and refuses an empty - #: one; `Walk.settle_boot_turn` waits it out before a word is typed. - boot_prompt: str | None + #: launch_arguments` puts its words last on the command line and refuses an + #: empty one; `Walk.settle_boot_turn` waits the turn out, through the reading + #: the value carries, before a word is typed. + boot: hand_started.BootPrompt | None #: The words that make this lane's agent spawn a Child Process. "subagent" #: and "sub-agent" appear here on purpose: this string is spoken *to* the #: agent, where it is the agent's own mechanism word and the thing that makes @@ -291,7 +293,7 @@ class Lane: # Launched silent. No boot gate of the Codex kind has been measured here — # `claude` boots into an empty composer — and a Session nobody has typed into # is what `roster` and `stable name`'s three reads want to find. - boot_prompt=None, + boot=None, relayed=lambda workspace: writing(RELAY_FILE, RELAY_WORD), # The flag the harness passes *is* the whole policy on this lane, and Claude # publishes no per-turn readback of it, so there is nothing to read back and @@ -355,7 +357,7 @@ class Lane: agent="codex", binary="codex", arguments=("--sandbox", "workspace-write"), - boot_prompt=ACKNOWLEDGE.words, + boot=hand_started.BootPrompt(words=ACKNOWLEDGE.words, turn_over=hand_started.codex_turn_over), # Measured 2026-08-27 through the shared daemon with the product's own pin # and no sandbox override, on codex-cli 0.149.1 and again on 0.150.0 over a # 0.149.1 app-server: a write to a path outside the workspace raises @@ -454,8 +456,8 @@ def walk(self) -> None: "the revoke. It is not a step: the run cannot both arrange this and judge it.", ) try: - self.arm_switches() self.settle_boot_turn() + self.arm_switches() except LaneBlocked as unarmed: self.journey.skip_rest(str(unarmed)) return @@ -479,31 +481,46 @@ def settle_boot_turn(self) -> None: Not a step, for `arm_switches`' reason: it is how this lane is put in the state the walk assumes, not a claim about the product. But it is not - optional either, and the failure it prevents is a **false green** rather - than a red. - - A lane with a `boot_prompt` is running a turn from the moment it starts - (#110 — the prompt is what carries it past the update gate). Nothing may - be typed into a Session that is mid-turn, and no chat mark may be taken - while one is in flight: `stable name` drives the walk's first turn and - hands `stop notice` the mark from just before it, so a boot turn that - ends *after* that mark puts its own Stop Notice on the far side of it — - and `stop notice` passes on a notice for a turn nobody drove, having + optional either, and what it prevents is a **false green** rather than a + red. + + A lane with a `boot` prompt is running a turn from the moment it starts + (#110 — a non-empty prompt is what carries it past the update gate). + Nothing may be typed into a Session that is mid-turn, and no chat mark may + be taken while one is in flight: `stable name` drives the walk's first + turn and hands `stop notice` the mark from just before it, so a boot turn + that ends *after* that mark puts its own Stop Notice on the far side of it + — and `stop notice` passes on a notice for a turn nobody drove, having proved nothing. The two turns carry the same words, so no reader of the chat could tell them apart afterwards either. - Over means what it means everywhere else here (`_turn_ended`): the - agent's own record grew and then stopped growing. Nothing is typed and - nothing is asked — this only watches. + **Two things close that, and both are needed.** This runs *before* + `arm_switches`, so every outlet is still off while the boot turn runs and + no notice for it can exist to be found (a fresh engine answers `duty off, + message off, voice off`, and Duty coming on reports only what is still + actionable — #80, which an idle Session is not). And it waits on the + agent's **own** turn boundary rather than on the record going quiet: + `_drive_turn` settles for nine seconds of silence, which for a graded turn + costs a slow reading and here would cost the run its meaning, because a + turn waiting on the model appends nothing either. Codex says which it is + (`hand_started.codex_turn_over`), so this asks Codex. + + Nothing is typed and nothing is asked — this only watches. """ - if self.lane.boot_prompt is None: + boot = self.lane.boot + if boot is None: return started = time.monotonic() - # Resolves the record this waits on. It is the ordinary first call: every - # step reads ground truth through here, and it is cached after the first. + # Resolves the record this waits on. The ordinary first call: every step + # reads ground truth through here, and it is cached after the first. truth = self._ground_truth() - deadline = started + self.far_side.agent_turn_seconds * BOOT_TURN_ALLOWANCE - turn = self._measured("boot prompt", started, self._turn_ended(deadline=deadline)) + allowed = self.far_side.agent_turn_seconds * BOOT_TURN_ALLOWANCE + over = support.wait_for( + lambda: boot.turn_over(self._record_now()), + deadline_seconds=allowed, + poll_seconds=2.0, + ) + turn = self._measured("boot prompt", started, bool(over)) if not turn.ended: raise LaneBlocked( f"the turn this lane was launched with had not ended after {turn.seconds:.0f}s, " @@ -513,9 +530,10 @@ def settle_boot_turn(self) -> None: ) self.journey.observe( "boot turn", - f"the Session was launched with {self.lane.boot_prompt!r} — the words `stable name` " - f"types anyway — because a non-empty prompt is what skips Codex's update gate " - f"(#110). Waited out before anything was typed: {turn.seconds:.1f}s, record " + f"the Session was launched with {boot.words!r} — the words `stable name` types " + f"anyway — because a non-empty prompt is what skips Codex's update gate (#110). " + f"Waited out on Codex's own `task_started`/`task_complete` bracketing, with every " + f"outlet still off, before anything was typed: {turn.seconds:.1f}s, record " f"{self._record_size()} bytes. Arranged by the harness, not judged by it.", ) @@ -1108,25 +1126,15 @@ def _drive_turn( """ started = time.monotonic() self.session.submit(instruction.words) - ended = self._turn_ended( - deadline=started + self.far_side.agent_turn_seconds, expect_waiting=expect_waiting - ) - return self._measured(what, started, ended) - - def _turn_ended(self, *, deadline: float, expect_waiting: bool = False) -> bool: - """Whether the turn in flight is over by `deadline`, read from outside. - - Shared by the turns the walk drives and the one the launch starts - (`settle_boot_turn`), because "over" has to mean the same thing for both: - a boot turn judged by a looser rule than the turns it runs ahead of is a - boot turn that can still be running when the next one starts. - """ settled_for = 0.0 last = self._record_size() + ended = False + deadline = started + self.far_side.agent_turn_seconds while time.monotonic() < deadline: time.sleep(3.0) if expect_waiting and self._roster_field("state") == "waiting": - return True + ended = True + break size = self._record_size() if size != last: last, settled_for = size, 0.0 @@ -1135,8 +1143,9 @@ def _turn_ended(self, *, deadline: float, expect_waiting: bool = False) -> bool: # A record that has not grown for two polls after growing at all is a # turn that is over; before it grows at all there is nothing to settle. if size > 0 and settled_for >= 9.0: - return True - return False + ended = True + break + return self._measured(what, started, ended) def _measured(self, what: str, started: float, ended: bool) -> Turn: """Record one turn on the walk, whoever asked for it.""" From fb1ac7ceb7a3a98f074e9da298c53983569669a5 Mon Sep 17 00:00:00 2001 From: Simon Qi Date: Thu, 27 Aug 2026 10:48:04 +1200 Subject: [PATCH 4/4] Drain the boot turn's own Stop Notice before marking the chat for a later one (#110 review) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Third review round, and the half of its P1 that was right: the rollout's `task_complete` and the engine's own observation of the Stop are two unsynchronised paths, so an engine that observes it *after* `arm_switches` escalates it straight out — and `stop notice`, whose mark is taken later, could then pass on a notice for a turn nobody drove. The commit before this one closed that by argument (the window is narrow, and the status is a push rather than the 5s poll), and a green that rests on a message losing a race is the thing this harness exists to refuse. Advisor's ruling, and the right one. `drain_boot_notice` waits on a mark taken behind the boot turn until the notice arrives or `telegram_round_trip_seconds` passes — the same window `stop notice` trusts for its own — so anything after `stable name`'s later mark is provably not the boot turn's. It **records and never asserts**, because *no notice* is a legitimate answer: a Stop is raised only on a transition out of `active`, and the first `idle` a thread reports is it sitting there having done nothing (`adapters/agent/codex/adapter.py:965-985`), so an engine that first saw this thread already idle raised nothing to drain. That case pays the worst case — one 90s wait per Codex run — and it is stated in `docs/acceptance-design.md` beside the boot turn with that reason. **The other half of the finding hurt nothing, and the reason is worth keeping.** A notice with no route open is retained rather than dropped (`core/escalation.py:230-232`) and `flip_switch(on=True)` sweeps the retained ones (`core/bridge.py:361-372`) — but that sweep happens *inside* `arm_switches`, which runs between the settle and the drain, so a held boot notice is delivered early, not late. The outlets-off half of the arrangement stands. The one `boot turn` observation now carries both halves: the seconds the turn took, whether its notice was drained and which chat message it was, and what `engine.log`'s `Session stopped` lines say — the same attribution `stop notice` makes for its own. Exercised before committing, all four paths: a notice that arrives is drained and named with the log line behind it; a notice that never comes is recorded as the legitimate case it is; a boot turn that never ends still blocks the lane with no mark taken and none spent; and the Claude lane does nothing at all. 1496 tests pass; ruff clean. Co-Authored-By: Claude Opus 5 (1M context) --- tests/acceptance/journey.py | 106 +++++++++++++++++++++++++++--------- 1 file changed, 79 insertions(+), 27 deletions(-) diff --git a/tests/acceptance/journey.py b/tests/acceptance/journey.py index b9b9a65..ab54814 100644 --- a/tests/acceptance/journey.py +++ b/tests/acceptance/journey.py @@ -94,6 +94,12 @@ #: measuring a Session with a turn still running underneath. BOOT_TURN_ALLOWANCE = 2.0 +#: The engine's own line for a Stop it announced (`core/bridge.py:_session_stopped`). +#: `stop notice` matches a looser pattern for its own purposes; `drain_boot_notice` +#: wants the announcement itself, because "was a notice raised for the boot turn" +#: is exactly the question it is asking. +ENGINE_STOP_LINE = r"(?i)Session stopped:" + #: What the escalated permission announcement says (`core/approvals.py:43-46`). #: Matched rather than quoted whole: the tool name and the detail are the agent's, #: and this run does not get to predict them. @@ -443,6 +449,9 @@ def __init__( self.before_first_turn: int | None = None #: Held for `approval`: what `relay`'s turn raised and how it was answered. self.approval_evidence: str | None = None + #: Held for `drain_boot_notice`, which writes the one `boot turn` + #: observation once both halves of the arrangement are done. + self.boot_turn: Turn | None = None self.turns: list[Turn] = [] # --- the walk --------------------------------------------------------- @@ -456,8 +465,9 @@ def walk(self) -> None: "the revoke. It is not a step: the run cannot both arrange this and judge it.", ) try: - self.settle_boot_turn() + boot_mark = self.settle_boot_turn() self.arm_switches() + self.drain_boot_notice(boot_mark) except LaneBlocked as unarmed: self.journey.skip_rest(str(unarmed)) return @@ -476,8 +486,8 @@ def walk(self) -> None: or "no turn ran", ) - def settle_boot_turn(self) -> None: - """Wait out the turn the *launch* started, before a word is typed at it. + def settle_boot_turn(self) -> int | None: + """Wait out the turn the *launch* started, and mark the chat behind it. Not a step, for `arm_switches`' reason: it is how this lane is put in the state the walk assumes, not a claim about the product. But it is not @@ -494,22 +504,19 @@ def settle_boot_turn(self) -> None: proved nothing. The two turns carry the same words, so no reader of the chat could tell them apart afterwards either. - **Two things close that, and both are needed.** This runs *before* - `arm_switches`, so every outlet is still off while the boot turn runs and - no notice for it can exist to be found (a fresh engine answers `duty off, - message off, voice off`, and Duty coming on reports only what is still - actionable — #80, which an idle Session is not). And it waits on the - agent's **own** turn boundary rather than on the record going quiet: - `_drive_turn` settles for nine seconds of silence, which for a graded turn - costs a slow reading and here would cost the run its meaning, because a - turn waiting on the model appends nothing either. Codex says which it is - (`hand_started.codex_turn_over`), so this asks Codex. - - Nothing is typed and nothing is asked — this only watches. + This waits on the agent's **own** turn boundary rather than on the record + going quiet. `_drive_turn` settles for nine seconds of silence, which for + a graded turn costs a slow reading and here would cost the run its + meaning, because a turn waiting on the model appends nothing either. + Codex says which it is (`hand_started.codex_turn_over`), so this asks + Codex. + + The mark it returns is the second half, and `drain_boot_notice` spends + it. Nothing is typed and nothing is asked — this only watches. """ boot = self.lane.boot if boot is None: - return + return None started = time.monotonic() # Resolves the record this waits on. The ordinary first call: every step # reads ground truth through here, and it is cached after the first. @@ -520,21 +527,66 @@ def settle_boot_turn(self) -> None: deadline_seconds=allowed, poll_seconds=2.0, ) - turn = self._measured("boot prompt", started, bool(over)) - if not turn.ended: + self.boot_turn = self._measured("boot prompt", started, bool(over)) + if not self.boot_turn.ended: raise LaneBlocked( - f"the turn this lane was launched with had not ended after {turn.seconds:.0f}s, " - f"so the walk cannot type into this Session without racing it. The agent " - f"reports {truth.describe()}; its record is {self._record_size()} bytes. " - f"Screen tail: {self.session.screen_tail()[-600:]!r}" + f"the turn this lane was launched with had not ended after " + f"{self.boot_turn.seconds:.0f}s, so the walk cannot type into this Session " + f"without racing it. The agent reports {truth.describe()}; its record is " + f"{self._record_size()} bytes. Screen tail: {self.session.screen_tail()[-600:]!r}" ) + return self.person.latest_message_id() + + def drain_boot_notice(self, mark: int | None) -> None: + """Let the boot turn's Stop Notice land before the walk marks the chat for a later one. + + **Turning an outlet on is what delivers what was held.** A notice with no + route open is retained rather than dropped (`core/escalation.py:230-232`), + and `flip_switch(on=True)` sweeps the retained ones + (`core/bridge.py:361-372`) — so `arm_switches`, which runs between + `settle_boot_turn` and this, is where a held boot notice goes out. That + much is already before `stable name`'s mark and needs nothing from here. + + What needs this is the other path. The rollout's `task_complete` and the + engine's own observation of the Stop are not synchronised, so an engine + that observes it after the arming escalates it straight out, and a green + `stop notice` would then rest on that message losing a race — which is + the one thing this harness may not do. So the walk waits here, on a mark + taken behind the boot turn, until either the notice arrives or the window + `stop notice` itself trusts has passed. + + **It records and never asserts**, because *no notice* is a legitimate + answer: a Stop is only raised on a transition out of `active`, and the + first `idle` a thread reports is it sitting there having done nothing + (`adapters/agent/codex/adapter.py:965-985`). An engine that first saw this + thread already idle raised nothing for the boot turn, and there is nothing + to drain — which is also the case that pays the full window. + """ + if mark is None or self.boot_turn is None: + return + arrived = self.person.await_message( + mark, deadline_seconds=self.far_side.telegram_round_trip_seconds + ) + announced = support.matching_lines(self.engine.log_lines(), ENGINE_STOP_LINE) + boot = self.lane.boot + assert boot is not None # there is no mark to spend without one self.journey.observe( "boot turn", - f"the Session was launched with {boot.words!r} — the words `stable name` types " - f"anyway — because a non-empty prompt is what skips Codex's update gate (#110). " - f"Waited out on Codex's own `task_started`/`task_complete` bracketing, with every " - f"outlet still off, before anything was typed: {turn.seconds:.1f}s, record " - f"{self._record_size()} bytes. Arranged by the harness, not judged by it.", + f"the Session was launched with {boot.words!r} — the words " + f"`stable name` types anyway — because a non-empty prompt is what skips Codex's " + f"update gate (#110). Waited out on Codex's own `task_started`/`task_complete` " + f"bracketing, with every outlet still off: {self.boot_turn.seconds:.1f}s. Its Stop " + f"Notice was then drained behind chat mark {mark} so that nothing after " + f"`stable name`'s later mark can be it: " + + ( + f"chat message {arrived.id} ({arrived.text[:80]!r})" + if arrived is not None + else f"nothing arrived in " + f"{self.far_side.telegram_round_trip_seconds:.0f}s, which is what an engine " + f"that first saw this thread already idle would do" + ) + + f"; engine.log Stop lines: {announced[-2:] or 'none'}. Arranged by the harness, " + f"not judged by it.", ) def arm_switches(self) -> None: