fix(supervision): keep Codex guarded through terminal catch-up - #1872
Open
DrWrong wants to merge 1 commit into
Open
fix(supervision): keep Codex guarded through terminal catch-up#1872DrWrong wants to merge 1 commit into
DrWrong wants to merge 1 commit into
Conversation
Initiating trigger: a Codex primary reached a retry Stop carrying stop_hook_active=true after the first turn-end guard block. Independent masking condition: the bounded foreground checkpoint had already exited, so a worker terminal status and turn-ended marker could arrive with no live watcher. Captain-visible symptom: the task finished in its pane but no durable wake callback continued the primary session. The earliest divergence from the proven Claude path was the default guard early-return on stop_hook_active=true. Claude treats that field only as continuation history and still requires watcher or auto-arm recovery proof; Codex incorrectly treated it as proof by itself. The smallest counterfactual changed only stop_hook_active on an otherwise unsupervised in-flight task: false blocked with exit 2, while true returned 0 before the fix. The end-to-end regression reproduced checkpoint exit, first Stop, terminal writes, retry Stop, late checkpoint catch-up, durable queue drain, and no replay. Disconfirming evidence retained: the watcher signal scanner already catches status and turn-ended writes that land while no watcher is live, and a late checkpoint surfaced them exactly once. The defect was therefore not missing durable scan semantics. The incident files had already been torn down, so the supplied timestamp evidence could not be re-read. Codex now rechecks supervision on every Stop. Grok retains its separate one-block contract through an explicit --grok mode; Claude, OpenCode, Pi/pi-signed, Kimi, and all runtime backends keep their existing integration boundaries.
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
Diagnose and fix the Firstmate supervision regression in which a Codex ship task can complete after the foreground checkpoint has exited and the captain receives no durable callback. Reproduce the end-user sequence: a Codex ship task remains active after the primary returns a final response; the foreground bin/fm-watch-checkpoint.sh cycle has exited; the primary turn attempts to end; the worker then writes terminal done status and state/.turn-ended with no live foreground checkpoint; and a checkpoint started later currently fails to surface that existing completion. The incident evidence is that state/pangle-rta-sampled-logs.turn-ended and terminal status were written at 2026-08-07 14:33 while state/.last-watcher-beat stopped at 14:16, and the captain saw the worker finish in its pane but received no callback or continuation. The Codex Stop hook must prevent a blind primary turn end or force a bounded continuation while tasks remain active. Explicitly separate the initiating trigger, independent masking condition, and captain-visible symptom; compare the failing Codex path with at least one proven guarded primary path and identify the earliest meaningful divergence in control flow or durable state handling; inspect the Stop hook, turn-end guard, checkpoint, wake queue, relevant history and tests without assuming the nearest change is causal; run the smallest counterfactual that should flip the result; name and seek disconfirming evidence, retaining contradiction or uncertainty that could alter scope; and record causal evidence in the commit and PR report without putting incident chronology in maintained product docs. Make terminal task events impossible to miss silently when a Codex foreground checkpoint is not live. Prefer deterministically enforcing the existing Stop-guard and durable catch-up contract over detached background processes. Ensure a late checkpoint or bounded session continuation surfaces a durable terminal event exactly once. Preserve durable queue authentication, primary lock ownership, task-event authentication, exactly-once acknowledgement, and every supported harness/backend protocol. Review every affected supported primary harness and runtime backend integration surface, marking an axis not applicable only after inspection. Do not modify or consume unrelated live task state, private captain data, or another task's durable records. Add executable/public-interface end-to-end coverage for active task -> checkpoint exits -> primary turn attempts to end -> worker writes terminal status and turn-ended -> captain notification durably surfaces exactly once; it must fail before and pass after, or precisely document why a pre-fix assertion cannot safely run and use the closest non-equivalent counterfactual. Run focused tests, bin/fm-lint.sh for shell changes, documentation audience check for maintained prose changes, and the repository no-mistakes pipeline. Open a PR and do not merge. Accepted implementation: Codex treats stop_hook_active as continuation history rather than recovery proof and rechecks the live supervision predicate on every Stop; Grok preserves its one-block behavior via explicit --grok; Claude, OpenCode, Pi, pi-signed, Kimi, and runtime backends retain their existing boundaries; existing watcher late-signal catch-up and authenticated durable queue mechanics remain unchanged. Add or refresh a real Codex live guard proving repeated Stop blocks are honored.
What Changed
stop_hook_activeas continuation history so terminal events cannot be missed after a foreground checkpoint exits.--grokguard mode while leaving other harness boundaries unchanged.Risk Assessment
✅ Low: The change is narrowly scoped and satisfies the accepted Codex repeated-Stop contract while explicitly preserving Grok’s one-block behavior and existing durable wake semantics.
Testing
Targeted guard and checkpoint tests exercised Codex’s retry-Stop supervision, Grok/Claude/OpenCode/Pi boundaries, and late authenticated terminal catch-up exactly once; a real Codex CLI 0.147.0 session also honored two consecutive Stop blocks, with reviewer-visible CLI logs captured. No screenshot was produced because the affected end-user surface is CLI/hook behavior rather than visual UI.
Evidence: Late-checkpoint exactly-once evidence
ok - late checkpoint surfaces a terminal event exactly once after the bounded Codex blind-stop continuationEvidence: Real Codex repeated-Stop evidence
ok - codex-cli 0.147.0 live E2E preserved the foreground checkpoint and honored two consecutive Stop blocksEvidence: Cross-harness turn-end guard evidence
Pipeline
Updates from git push no-mistakes
✅ **intent** - passed
✅ No issues found.
✅ **Rebase** - passed
✅ No issues found.
✅ **Review** - passed
✅ No issues found.
✅ **Test** - passed
✅ No issues found.
Inspectedgit diff 70aeba855527f7693082f6dd1bc731e334d0269f..210efdc5ed22ed85377305511d25e936410164deand the target commit’s causal evidencetests/fm-turnend-guard.test.shtests/fm-watch-checkpoint.test.shFM_CODEX_LIVE_E2E=1 tests/fm-codex-continuity-live-e2e.test.shgit status --shortafter transient test cleanup✅ **Document** - passed
✅ No issues found.
✅ **Push** - passed
✅ No issues found.