fix(watch): stop escalating a decision-gate run whose worker is alive as a wedge - #82
Merged
Merged
Conversation
added 4 commits
August 9, 2026 03:09
…is alive The fleet already knew. On 2026-08-08 two workers escalated as "idle 254s, possible wedge, escalation 1" while bin/fm-crew-state.sh, run against the same tasks at the same moment, returned "state: parked - source: run-step - parked at review: 3 finding(s) (ask-user: captain decision)". Both panes were idle because the worker had asked its question and stopped, exactly as its brief requires. crew_absorb_class collapsed that authoritative `parked` reading into `none`, so the stale path had no way to tell "stopped on purpose at a gate" from "stopped, reason unknown" and climbed the escalation ladder against it. `parked` is now its own absorb class, restricted to source: run-step - a status-log-sourced gate is an arbitrarily old append-only EVENT and reads the same whether the crew is at the gate or died an hour ago. The class alone never absorbs anything. A worker that crashed one second after printing its gate prompt leaves the run parked identically and forever, so the watcher corroborates with the one signal that changes when the worker dies: the agent PROCESS, via the same fm_backend_agent_alive probe the codex static-pane backstop uses. Only `alive` holds the ladder; `dead` and `unknown` fall through to the unchanged surface and the unchanged escalation timings. The liveness reading is re-taken at every escalation window rather than trusted once, and the hold still earns one bounded recheck per FM_PAUSE_RESURFACE_SECS, so a gate nobody ever answers cannot rot invisibly. Deliberately out of scope, and left at today's behavior: - pi (its launcher execs into a generic `node`) and every backend past tmux/herdr read `unknown`, so they get no hold at all. - The declared-pause and captain-held branch keeps `parked` mapped to `none`; its own dead-agent rule already owns that idle pane. - bin/fm-mark-parked.sh is untouched. A mid-pipeline gate is neither terminal nor an external wait, and a hand-placed marker would keep muting the pane after the gate was answered. Costs no extra crew-state read: both call sites already held the verdict, and only the `parked` branch adds a liveness probe. Nothing new runs per poll. The two hold sites are now one function, ladder_hold, so the timer refresh, the frozen-hash age anchor, and the throttle marker cannot drift apart. Tests cover all four states plus both failure directions: parked+alive absorbed with the wedge timer armed, parked+dead surfaced immediately, parked+dead escalating on the unchanged ladder schedule, unknown liveness surfacing as today, and a status-log-sourced gate earning no class.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
Stop the firstmate supervisor raising possible-wedge alarms for workers that are demonstrably fine, without making it blind to workers that are genuinely stuck.
Observed defect: state/.wake-queue repeatedly carried lines like 'stale: coditan:fm- (idle 254s, possible wedge, escalation 1)' while bin/fm-crew-state.sh, run against the same task at the same moment, returned the definitive authoritative answer 'state: parked - source: run-step - parked at review: 3 finding(s) (ask-user: captain decision)'. The fleet already knew the worker was parked waiting on a decision and alarmed anyway. Root cause: crew_absorb_class in bin/fm-classify-lib.sh collapsed 'state: parked' into the 'none' token, so the watcher's stale path could not tell 'stopped on purpose at a gate' from 'stopped, reason unknown', and climbed the wedge escalation ladder against it.
The hard half, and the explicit design constraint the captain set: a wedge alarm that stops firing is WORSE than one that fires too often, because nobody notices the silence. A worker parked at a gate looks identical to a worker that crashed just after printing the gate prompt - same idle pane, same last status line, and no-mistakes keeps reporting the run as parked forever either way. So suppressing alarms for anything reporting 'parked' was explicitly ruled out; the fix had to be able to say WHY it believes the worker is fine, from evidence that would change if the worker died.
Deliberate decisions made while doing the work:
Tests cover all four states and both failure directions, not just the happy path: parked+alive absorbed with the wedge timer still armed, parked+dead surfaced immediately, parked+dead still escalating on the unchanged ladder schedule, unknown liveness surfacing as today, and a status-log-sourced gate earning no absorb class. The three existing codex-backstop tests are untouched and still pass.
Verified before submitting: bin/fm-lint.sh full sweep rc=0 with no findings, tests/fm-watch-triage.test.sh 63/63, tests/fm-crew-state.test.sh 73/73, tests/fm-daemon.test.sh 97/97, tests/fm-instruction-owners.test.sh rc=0.
What Changed
crew_absorb_classinbin/fm-classify-lib.shnow returns a distinctparkedtoken when the authoritative crew state reports a run stopped at a no-mistakes decision gate, restricted tosource: run-step; a gate known only from the append-only status log still collapses tonone.bin/fm-watch.shaddsparked_gate_liveness_class, which only honors that token whenfm_backend_agent_aliveconfirms the worker's agent process (dead/unknownfall through to today's surface and escalation timings, secondmates excluded). The wedge-escalation path re-reads liveness at every escalation window and, when it holds, routes through a new sharedladder_holdhelper that both the active-run and parked-gate holds use for timer refresh, the frozen-hash-age bounded recheck, and the.wedgeheldmarker. Aparkedclass short-circuits tononeon the first-sighting path beforecodex_static_pane_upgrade(so codex gets no first-sight absorb the other harnesses lack) and in the declared-pause branch ofpause_state_class, leaving both unchanged.tests/fm-watch-triage.test.shwith coverage for parked+alive absorbed with the timer still armed, parked+dead surfaced and still escalating on the unchanged ladder, unknown liveness surfacing as before, the bounded recheck for a long-held gate, and a status-log-sourced gate earning no absorb class; documents the rule indocs/architecture.md,docs/configuration.md(FM_STALE_ESCALATE_SECS,FM_PAUSE_RESURFACE_SECS), the harness-adapters skill, and a boundary comment inbin/fm-mark-parked.sh.Risk Assessment
✅ Low: The change is well-bounded to one absorb class and one escalation branch, preserves the existing wake text and escalation timings byte-for-byte, and is covered by regression tests in both failure directions (parked+dead and parked+unknown still escalate on the unchanged 240s ladder, and the codex first-sight test genuinely fails against the prior commit); the only open item is a stale clause in the intent text describing codex backstop coverage that the captain already authorized narrowing.
Testing
I reproduced the reported defect by driving the real watcher over a crew whose authoritative run-step reports it parked at an ask-user review gate: at the base commit it queues
stale: … (idle 500s, possible wedge, escalation 1), and at the target commit it queues nothing and records in the triage log why it believes the worker is fine. The same transcript shows the crashed-worker case still raising the identical wedge alarm at both commits, so the alarm did not go silent. I ran each parked-gate test individually against a base-code tree to confirm the five defect tests fail before the fix and pass after, while the three dead/unknown-liveness tests pass in both directions, which is what demonstrates the escalation cadence is unchanged. The watch-triage, crew-state, and daemon suites all pass. The one claimed behavior with no automated coverage was the parked branch's bounded recheck — the guarantee that a gate nobody ever answers still reaches the captain — so I added a focused test for it that fails at base and passes at target. This is a CLI supervisor with no rendered UI surface, so the reviewer-visible evidence is captured as watcher and wake-queue transcripts rather than screenshots.Evidence: Before/after: what reaches the captain's wake queue for a parked gate (pre-fix)
$ bin/fm-crew-state.sh fm-review3 # the authoritative answer, at this moment state: parked · source: run-step · parked at review: 3 finding(s) (ask-user: captain decision) $ bin/fm-watch.sh # one supervisor pass over the fleet stale: crew:fm-review3 (idle 500s, possible wedge, escalation 1) $ cat state/.wake-queue # what reaches the captain 1786252462 1 stale crew:fm-review3 stale: crew:fm-review3 (idle 500s, possible wedge, escalation 1) $ cat state/.wedge-escalations-crew_fm-review3 # the wedge ladder position 1Evidence: After the fix: alive worker held with a stated reason, crashed worker still alarms, unanswered gate still gets its bounded recheck
CASE: worker PARKED at the gate and ALIVE - the captain's reported false alarm $ cat state/.wake-queue (empty - no alarm raised) $ cat state/.wedge-escalations-crew_fm-review3 (absent - the ladder never climbed) $ tail -1 state/.watch-triage.log […] absorbed stale (overridden terminal status) (the run is parked at a decision gate and this worker is still alive, ladder held): crew:fm-review3 CASE: same parked gate, but the worker CRASHED $ cat state/.wake-queue 1786252468 1 stale crew:fm-review3dead stale: crew:fm-review3dead (idle 501s, possible wedge, escalation 1) CASE: parked and alive, but nobody has answered the gate for a full recheck window $ cat state/.wake-queue 1786252468 1 stale crew:fm-review3rot stale: crew:fm-review3rot (pane unchanged 500s while the run is parked at a decision gate and this worker is still alive - bounded recheck on a long cadence, not a wedge escalation; answer the gate or confirm the decision is still pending)Evidence: Per-test regression proof: same tests, base code vs target code
### test_parked_gate_alive_holds_the_wedge_ladder BEFORE (base e6762c5): not ok - a live worker parked at an ask-user gate still escalated as a possible wedge: stale: test:fm-ladderalive (idle 500s, possible wedge, escalation 1) AFTER (target 0c6c719): ok - a live worker parked at an ask-user gate holds the wedge ladder instead of escalating ### test_parked_gate_hold_gets_bounded_recheck BEFORE (base e6762c5): not ok - the recheck did not identify itself as a bounded recheck: stale: test:fm-gaterot (idle 501s, possible wedge, escalation 1) AFTER (target 0c6c719): ok - a parked gate nobody answers still earns one bounded recheck per window, never a wedge escalation ### test_parked_gate_dead_escalates_on_the_ladder BEFORE (base e6762c5): ok - a run parked at a gate whose worker died escalates on the unchanged wedge schedule AFTER (target 0c6c719): ok - a run parked at a gate whose worker died escalates on the unchanged wedge schedule ### test_parked_gate_unknown_liveness_escalates_on_the_ladder BEFORE (base e6762c5): ok - a parked worker whose agent liveness cannot be read keeps today's escalation AFTER (target 0c6c719): ok - a parked worker whose agent liveness cannot be read keeps today's escalationPipeline
Updates from git push no-mistakes
✅ **intent** - passed
✅ No issues found.
✅ **Rebase** - passed
✅ No issues found.
bin/fm-watch.sh:1813- The new first-sightparkedabsorb suppresses the immediate surface for a gate that was never relayed to firstmate. This branch is only reachable through the non-terminal path (line 1784), which is entered exactly whenstale_is_terminalis false — i.e. the crew's last status line is NOT captain-relevant, so noneeds-decision:/blocked:was ever appended. In that state the run-step saysparked at reviewandfm_backend_agent_alivesaysalive(aclaudepane sitting idle at its prompt after a turn ends reportsalive), so the wake is absorbed silently and the only further signal isladder_hold's bounded recheck at FM_PAUSE_RESURFACE_SECS (3600s default). The reported defect does not need this branch: the observedidle 254s, possible wedge, escalation 1lines come from a crew whoseneeds-decision:WAS relayed, which takes the terminal-override path (1748) and reacheswedge_timer_check— and that site's newparkedhold (line 461) already fixes it. The terminal path guards its equivalent absorb onstatus_already_surfaced(line 1749); the non-terminalparkedbranch has no such corroboration, so it absorbs precisely the case where the decision has not reached anyone. Suggested earliest boundary: gate the first-sightparkedabsorb on the gate having already been surfaced (the existingstatus_already_surfaced "$STATE/$task.status"check), and fall through tosurface_nonterminal_staleotherwise; the ladder hold at 461 is unaffected either way. Flagging rather than fixing because the intent explicitly specifies "absorb and start the wedge timer exactly asworkingdoes" and tests/fm-watch-triage.test.sh:422 (test_parked_gate_alive_absorbed, statusworking: running the pipeline, asserting no wake is queued) enshrines it — this is the author's deliberate choice and the tradeoff against the captain's "silence is worse than noise" constraint is theirs to make.bin/fm-supervise-daemon.sh:997- The away-mode path is untouched, so the same demonstrably-fine worker still produces a possible-wedge alarm whilestate/.afkexists. In away mode the watcher short-circuits at bin/fm-watch.sh:1724 and enqueues a barestale: $wwithout consultingcrew_absorb_classat all; the away daemon's stale-persistence recheck then escalatesstale persisted ${age}s (possible wedge): $winpurely from a pane-busy re-peek, with no run-step or liveness input. Noting it only so the coverage boundary is explicit — the daemon is a separate triage owner with a different escalation consumer, the reported evidence is the watcher's own(idle Ns, possible wedge, escalation N)format, and the intent scopes the change to the watcher. No action expected in this change.🔧 Fix: surface a parked gate's first sighting instead of absorbing it
2 issues (1 warning, 1 info) still open:
bin/fm-watch.sh:842- The first-sight parked absorb was removed for every harness except codex.pause_state_classmapsparkedtonone(line 841) and then feeds thatnoneintocodex_static_pane_upgrade(line 842), whose only precondition isfallback = none+harness = codex+kind != secondmate+ a live agent process (lines 766-772). A codex worker parked at a gate satisfies all four, so the upgrade returnsworking, the main loop absorbs it as "provably working" and arms the wedge timer (line 1805-1809), and every subsequent escalation window then reaches the newparkedbranch and holds the ladder (line 456-467). Concrete path: harness=codex,fm-crew-state.shreturnsstate: parked - source: run-step - parked at review: N finding(s) (ask-user: captain decision), and the crew's last status line is NOT captain-relevant (that is what routes the poll to this branch), so no needs-decision/blocked line ever reached firstmate. At base e6762c5 that pane escalated at FM_STALE_ESCALATE_SECS (240s) aspossible wedge, escalation 1; with this change nothing reaches firstmate untilladder_hold's bounded recheck at FM_PAUSE_RESURFACE_SECS (3600s default). That is a 15x delay on an unrelayed captain decision for one harness, and it contradicts the invariant this fix round was told to establish ("a first sighting always surfaces") and the file's own new comment at lines 835-840. Earliest supported shared boundary: in the undeclared branch ofpause_state_class, returnnonedirectly for aparkedclass instead of routing it throughcodex_static_pane_upgrade- the codex backstop exists for a healthy codex worker mid-turn on a static pane (its own comment, lines 735-742), which is a run-stepworking/no-run-step case, not a run the authoritative run-step reports stopped at a gate, so its documented coverage is not reduced. Raising rather than fixing because the intent states "The codex streaming backstop (codex_static_pane_upgrade) was deliberately left intact ... Its coverage is unchanged", and narrowing it is the author's call.bin/fm-watch.sh:776- Informational, for whoever writes the PR body: the intent text still describes the pre-fix-round shape and no longer matches the code. It says "Both consumers corroborate it with fm_backend_agent_alive" and "The two ladder-hold sites were factored into one function" - there is now exactly one consumer of the gate class (wedge_timer_check), and the twoladder_holdsites are the working hold and the parked hold, both inside that one function. It also lists "parked+alive absorbed with the wedge timer still armed" and "unknown liveness surfacing as today" as the test coverage; the fix round replaced those withtest_parked_gate_first_sight_surfaces_then_holds_the_ladder(first sighting surfaces, hold applies from the second sighting) andtest_parked_gate_unknown_liveness_escalates_on_the_ladder. The code and tests are self-consistent; only the narrative is stale. No action needed on the implementation.🔧 Fix: never route a parked gate through the codex backstop
1 info still open:
bin/fm-watch.sh:847- The shipped code narrows codex_static_pane_upgrade's reach, which contradicts the intent's clause: "The codex streaming backstop (codex_static_pane_upgrade) was deliberately left intact and NOT applied at the wedge-escalation site... Its coverage is unchanged." At base e6762c5, crew_absorb_class collapsed a run-step gate intonone, so a codex worker parked at a gate DID reach that backstop and, with a live codex process, was absorbed as provably working. Commit 0c6c719 added the short-circuit at bin/fm-watch.sh:847-850, which returnsnonebefore the backstop runs, so that same pane now surfaces astale:wake on first sight. Concrete case: harness=codex, no-mistakes reportsawaiting_agent:/fix_reviewso fm-crew-state.sh returnsstate: parked - source: run-step, and codex is mid-tool-call on a static pane (the exact false-idle blind spot documented at bin/fm-watch.sh:731-742) - base absorbed it, HEAD surfaces it. The impact is bounded (one wake per distinct static hash, not a wedge escalation, and every non-codex harness already behaved this way at base), and every subsequent sighting still holds the ladder via the new parked branch. This is exactly what round 2's instruction directed ("return none directly for a parked class, before codex_static_pane_upgrade"), and the in-code comment at 835-846 plus docs/architecture.md were correctly updated to describe it - only the --intent text still asserts the old claim. Raising for the record because the intent argument is authoritative acceptance criteria; no code change is needed unless you'd rather restore codex's first-sight absorb, in which case the intent's "coverage is unchanged" clause would hold instead.✅ **Test** - passed
✅ No issues found.
bash tests/fm-watch-triage.test.sh— full suite, rc=0, 65 ok (includes the 6 new parked-gate tests plus the one I added)bash tests/fm-crew-state.test.sh— rc=0, 73 ok (the other crew_absorb_class consumer)bash tests/fm-daemon.test.sh— rc=0, 97 ok (fm-classify-lib.sh consumer)Regression proof: each parked-gate test run in its own process against a tree withbin/fm-classify-lib.sh+bin/fm-watch.shreverted to base e6762c5, then against target 0c6c719 —test_crew_absorb_class_classifier,test_parked_gate_first_sight_surfaces_then_holds_the_ladder,test_parked_gate_codex_first_sight_surfaces_then_holds_the_ladder,test_parked_gate_alive_holds_the_wedge_ladder,test_parked_gate_hold_gets_bounded_recheckfail before / pass afterUnchanged-behavior proof:test_parked_gate_dead_surfaces,test_parked_gate_dead_escalates_on_the_ladder,test_parked_gate_unknown_liveness_escalates_on_the_ladderpass identically at base and targetManual end-to-end: drove the realbin/fm-watch.shover a parked-at-review-gate crew (alive worker, crashed worker, and an unanswered gate held a full recheck window) and capturedstate/.wake-queue,state/.wedge-escalations-<key>, andstate/.watch-triage.logat both commitsNew test added:test_parked_gate_hold_gets_bounded_recheckintests/fm-watch-triage.test.sh— covers the parked branch's bounded recheck, throttle marker, frozen-hash age anchor, and no-ladder-climbgit status --porcelain— worktree carries only the intended test addition.agents/skills/afk/SKILL.md:165- Scope judgment call, no edit made: the away-mode daemon (bin/fm-supervise-daemon.sh) was deliberately left unchanged, so while state/.afk is present a worker parked at a decision gate still ages into a possible-wedge escalation on the housekeeping path. .agents/skills/afk/SKILL.md:165 documents that behavior and remains factually correct, and docs/architecture.md scopes the new hold to the watcher, so nothing there is stale. Flagging only because the two paths now disagree on the same pane; if the asymmetry is meant to be permanent it may be worth one line in the afk skill naming it, and if it is meant to close, that is a follow-up code change rather than a docs one.✅ **Lint** - passed
✅ No issues found.
✅ **Push** - passed
✅ No issues found.