Skip to content

fix(supervision): keep Codex guarded through terminal catch-up - #1872

Open
DrWrong wants to merge 1 commit into
kunchenguid:mainfrom
DrWrong:fm/codex-terminal-catchup
Open

fix(supervision): keep Codex guarded through terminal catch-up#1872
DrWrong wants to merge 1 commit into
kunchenguid:mainfrom
DrWrong:fm/codex-terminal-catchup

Conversation

@DrWrong

@DrWrong DrWrong commented Aug 7, 2026

Copy link
Copy Markdown

Intent

Diagnose and fix the Firstmate supervision regression in which a Codex ship task can complete after the foreground checkpoint has exited and the captain receives no durable callback. Reproduce the end-user sequence: a Codex ship task remains active after the primary returns a final response; the foreground bin/fm-watch-checkpoint.sh cycle has exited; the primary turn attempts to end; the worker then writes terminal done status and state/.turn-ended with no live foreground checkpoint; and a checkpoint started later currently fails to surface that existing completion. The incident evidence is that state/pangle-rta-sampled-logs.turn-ended and terminal status were written at 2026-08-07 14:33 while state/.last-watcher-beat stopped at 14:16, and the captain saw the worker finish in its pane but received no callback or continuation. The Codex Stop hook must prevent a blind primary turn end or force a bounded continuation while tasks remain active. Explicitly separate the initiating trigger, independent masking condition, and captain-visible symptom; compare the failing Codex path with at least one proven guarded primary path and identify the earliest meaningful divergence in control flow or durable state handling; inspect the Stop hook, turn-end guard, checkpoint, wake queue, relevant history and tests without assuming the nearest change is causal; run the smallest counterfactual that should flip the result; name and seek disconfirming evidence, retaining contradiction or uncertainty that could alter scope; and record causal evidence in the commit and PR report without putting incident chronology in maintained product docs. Make terminal task events impossible to miss silently when a Codex foreground checkpoint is not live. Prefer deterministically enforcing the existing Stop-guard and durable catch-up contract over detached background processes. Ensure a late checkpoint or bounded session continuation surfaces a durable terminal event exactly once. Preserve durable queue authentication, primary lock ownership, task-event authentication, exactly-once acknowledgement, and every supported harness/backend protocol. Review every affected supported primary harness and runtime backend integration surface, marking an axis not applicable only after inspection. Do not modify or consume unrelated live task state, private captain data, or another task's durable records. Add executable/public-interface end-to-end coverage for active task -> checkpoint exits -> primary turn attempts to end -> worker writes terminal status and turn-ended -> captain notification durably surfaces exactly once; it must fail before and pass after, or precisely document why a pre-fix assertion cannot safely run and use the closest non-equivalent counterfactual. Run focused tests, bin/fm-lint.sh for shell changes, documentation audience check for maintained prose changes, and the repository no-mistakes pipeline. Open a PR and do not merge. Accepted implementation: Codex treats stop_hook_active as continuation history rather than recovery proof and rechecks the live supervision predicate on every Stop; Grok preserves its one-block behavior via explicit --grok; Claude, OpenCode, Pi, pi-signed, Kimi, and runtime backends retain their existing boundaries; existing watcher late-signal catch-up and authenticated durable queue mechanics remain unchanged. Add or refresh a real Codex live guard proving repeated Stop blocks are honored.

What Changed

  • Recheck live supervision on every Codex Stop, treating stop_hook_active as continuation history so terminal events cannot be missed after a foreground checkpoint exits.
  • Preserve Grok’s one-block behavior through an explicit --grok guard mode while leaving other harness boundaries unchanged.
  • Add deterministic and live coverage for repeated Codex Stop blocks and exactly-once late checkpoint catch-up, with corresponding supervision documentation updates.

Risk Assessment

✅ Low: The change is narrowly scoped and satisfies the accepted Codex repeated-Stop contract while explicitly preserving Grok’s one-block behavior and existing durable wake semantics.

Testing

Targeted guard and checkpoint tests exercised Codex’s retry-Stop supervision, Grok/Claude/OpenCode/Pi boundaries, and late authenticated terminal catch-up exactly once; a real Codex CLI 0.147.0 session also honored two consecutive Stop blocks, with reviewer-visible CLI logs captured. No screenshot was produced because the affected end-user surface is CLI/hook behavior rather than visual UI.

Evidence: Late-checkpoint exactly-once evidence

ok - late checkpoint surfaces a terminal event exactly once after the bounded Codex blind-stop continuation

ok - quiet checkpoint exits 124 with a clean checkpoint line and no live lock
ok - checkpoint passes through a real watcher wake and leaves the queue for drain
ok - checkpoint preserves watcher environment for registered custom checks
ok - checkpoint rejects an existing watcher singleton as unowned
●━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
●  WATCHER DOWN - SUPERVISION IS OFF
●  1 task(s) in flight, but no live watcher process holds this home lock (last beat: 1s ago).
●  Trust the emitted supervision protocol for this harness; do not use shell & for watcher repair.
●  This is a supervision warning only; the guarded operation WILL still run.
●  repair missing watcher supervision according to the session-start block for this harness; do not use shell &.
●━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
ok - late checkpoint surfaces a terminal event exactly once after the bounded Codex blind-stop continuation
Evidence: Real Codex repeated-Stop evidence

ok - codex-cli 0.147.0 live E2E preserved the foreground checkpoint and honored two consecutive Stop blocks

ok - codex-cli 0.147.0 live E2E preserved the foreground checkpoint and honored two consecutive Stop blocks
Evidence: Cross-harness turn-end guard evidence
ok - fm_supervision_unhealthy: false with no state/*.meta at all
ok - fm_supervision_unhealthy: true with in-flight task and no beacon ever
ok - fm_supervision_unhealthy: true with in-flight task and a beacon far outside the grace window
ok - fm_supervision_unhealthy: false with in-flight task and a fresh beacon
ok - fm_supervision_status: FM_SUP_QUEUE_PENDING tracks state/.wake-queue
ok - fm_supervision_needed: X-mode relay poll needs supervision
ok - fm_supervision_unhealthy: source-only home needs supervision
ok - fm-turnend-guard: silent no-op with nothing in flight
ok - fm-turnend-guard: blocks when a fresh beacon has no live watcher lock
ok - fm-turnend-guard: non-Claude path blocks a source-only home
ok - fm-turnend-guard: blocks on a dead watcher lock even when the beacon is fresh
ok - fm-turnend-guard: silent no-op with a live watcher lock and fresh beacon
ok - fm-turnend-guard: healthy non-Claude harness paths ignore Claude episode contention
ok - fm-turnend-guard: blocks on a live watcher lock with an ancient beacon
ok - fm-turnend-guard: blocks with the exact required reason in the primary when unhealthy
ok - fm-turnend-guard: blocks from active FM_HOME state, not only repo-root state
ok - fm-turnend-guard: X-mode repair reason sources the cadence config
ok - fm-turnend-guard: X-mode-only supervision remains guarded in default mode
ok - fm-turnend-guard: ignores stale repo-root state when FM_HOME is set
ok - fm-turnend-guard: uses FM_STATE_OVERRIDE ahead of FM_HOME/state
ok - fm-turnend-guard: Codex stop_hook_active continuation re-checks the live supervision predicate
ok - fm-turnend-guard: blocks a blind turn end in a secondmate's own home (.fm-secondmate-home no longer excludes it)
ok - fm-turnend-guard: idle-by-default - silent in a secondmate home with nothing in flight
ok - fm-turnend-guard: secondmate Codex continuation re-checks the live supervision predicate
ok - fm-turnend-guard: secondmate deferred-death recovery - silent while watched, forces re-arm once the watcher exits
ok - fm-turnend-guard: inert in a secondmate's own child worktree (linked git worktree) even when unhealthy
ok - fm-turnend-guard: blocks a blind turn end in a treehouse-leased LINKED secondmate home (marker force-include)
ok - fm-turnend-guard: an invalid (empty) marker cannot spoof inclusion; linked worktree stays exempt
ok - fm-turnend-guard: a non-ASCII marker cannot spoof inclusion; linked worktree stays exempt
ok - fm-turnend-guard: inert in a crewmate/scout task worktree (linked git worktree) even when unhealthy
ok - fm-turnend-guard: fails open (never blocks) when jq is missing
ok - fm-turnend-guard: silent no-op on empty stdin
ok - fm-turnend-guard: runs well under the generous timing margin (0s)
ok - fm-turnend-guard-grok: forces one explicitly marked same-session resume when the shared predicate blocks
ok - fm-turnend-guard-grok: legacy environment loop guard prevents a nested resume loop
ok - fm-turnend-guard-grok: native false delegates blocking feedback with zero resume processes
ok - fm-turnend-guard-grok: native true remains bounded and starts no resume process
ok - fm-turnend-guard-grok: both spellings are typed and camelCase has deterministic precedence
ok - fm-turnend-guard-grok: malformed, invalidly typed, and missing-prerequisite payloads start neither path
ok - fm-turnend-guard-grok: missing jq and no-supervision-needed stops stay silent and bounded
ok - .codex/hooks.json: Stop hook uses hook process root when payload cwd is outside
ok - .codex/hooks.json: Stop hook ignores nested git root guard scripts
ok - .opencode primary plugin: guard path is anchored to worktree, not directory
ok - .pi primary extension: no-tool and multi-tool runs each inject exactly one guard follow-up
ok - .pi primary extension: delivery failure resets the logical-run latch
ok - fm-turnend-guard --claude: re-blocks a loop-guarded stop while unhealthy and unclaimed (incident regression)
ok - fm-turnend-guard --claude: X-mode-only homes re-block when auto-arm recovery is absent
ok - fm-turnend-guard --claude: a live arming epoch advances once and repeated observation is idempotent
ok - fm-turnend-guard --claude: repeated failed-to-arming races make bounded monotonic progress
ok - fm-turnend-guard --claude: terminal owner boundary excludes a concurrent start without deadlock
ok - fm-turnend-guard --claude: fresh rewake epoch prevents a duplicate continuation for the same event
ok - fm-turnend-guard --claude: fresh failed epochs preserve and advance monotonic fail-open progression
ok - fm-turnend-guard --claude: integrated fresh failures reach one bounded fail-open, stop continuation, and reset on recovery
ok - fm-turnend-guard --claude: reset contention preserves all episode state until retry
ok - fm-turnend-guard --claude: concurrent auto-arm and guard resets are idempotent and deadlock-free
ok - fm-turnend-guard --claude: stale rewake epoch does not allow a blind stop
ok - fm-turnend-guard --claude: budget exhaustion alone cannot permit a blind stop
ok - fm-turnend-guard --claude: verified fail-open is loud, bounded, attended, and non-repeating
ok - fm-turnend-guard --claude: fail-open requires both exhausted retries and consumed notice
ok - fm-turnend-guard --claude: away ownership excludes the Stop-autoarm fail-open
ok - fm-turnend-guard --claude: positive watcher recovery resets failure episode state
ok - fm-turnend-guard --claude: bounded claim wait avoids a token-consuming forced continuation
ok - fm-turnend-guard --claude: secondmate home re-blocks unclaimed and allows auto-arm-claimed stops

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

✅ **Review** - passed

✅ No issues found.

✅ **Test** - passed

✅ No issues found.

  • Inspected git diff 70aeba855527f7693082f6dd1bc731e334d0269f..210efdc5ed22ed85377305511d25e936410164de and the target commit’s causal evidence
  • tests/fm-turnend-guard.test.sh
  • tests/fm-watch-checkpoint.test.sh
  • FM_CODEX_LIVE_E2E=1 tests/fm-codex-continuity-live-e2e.test.sh
  • git status --short after transient test cleanup
✅ **Document** - passed

✅ No issues found.

⚠️ **Lint** - 1 warning
  • ⚠️ linter found issues (exit code 127)
✅ **Push** - passed

✅ No issues found.

Initiating trigger: a Codex primary reached a retry Stop carrying stop_hook_active=true after the first turn-end guard block.

Independent masking condition: the bounded foreground checkpoint had already exited, so a worker terminal status and turn-ended marker could arrive with no live watcher.

Captain-visible symptom: the task finished in its pane but no durable wake callback continued the primary session.

The earliest divergence from the proven Claude path was the default guard early-return on stop_hook_active=true. Claude treats that field only as continuation history and still requires watcher or auto-arm recovery proof; Codex incorrectly treated it as proof by itself.

The smallest counterfactual changed only stop_hook_active on an otherwise unsupervised in-flight task: false blocked with exit 2, while true returned 0 before the fix. The end-to-end regression reproduced checkpoint exit, first Stop, terminal writes, retry Stop, late checkpoint catch-up, durable queue drain, and no replay.

Disconfirming evidence retained: the watcher signal scanner already catches status and turn-ended writes that land while no watcher is live, and a late checkpoint surfaced them exactly once. The defect was therefore not missing durable scan semantics. The incident files had already been torn down, so the supplied timestamp evidence could not be re-read.

Codex now rechecks supervision on every Stop. Grok retains its separate one-block contract through an explicit --grok mode; Claude, OpenCode, Pi/pi-signed, Kimi, and all runtime backends keep their existing integration boundaries.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant