CI mirror: fix(supervision): suppress false-positive stale wakes and catch runs parked at a gate - #6
Open
sbracewell64 wants to merge 9 commits into
Open
CI mirror: fix(supervision): suppress false-positive stale wakes and catch runs parked at a gate#6sbracewell64 wants to merge 9 commits into
sbracewell64 wants to merge 9 commits into
Conversation
Six paths emit a "stale:" wake - the watcher's surface_nonterminal_stale,
handle_paused_stale, wedge_timer_check, terminal-status override and away-mode
one-shot, plus the push fast-path - and all six rendered the same bare
"stale: <window>" opening. Nothing downstream could tell which rule fired, so
per-branch wake attribution had to be reconstructed from truncation patterns in
the absorbed-wake log; that method left 44% of stale wakes unattributed across
two independent readings of this system.
Every stale reason now ends with a "[branch=<name>]" tag, and the arm layer
carries it into the cycle-exit ledger as reason=actionable-stale-<branch>, which
is the existing durable per-wake record - no second collector is added. The tag
is prose and a substring match, never a parsed field.
Decorating every stale reason exposed a latent defect: bin/fm-supervise-daemon.sh
backs a window out of the reason with a bare "${reason#stale: }" prefix strip on
every away-mode wake, which was already silently wrong for the decorated wedge
and pause reasons - it yields "<window> (idle 300s, ...)", matches no recorded
window, and falls through window_to_task's suffix heuristic to a garbage task id.
Extraction moves to one owner, stale_reason_window, which strips decoration.
Two further measurements ride the same two logs, both sampled rather than
continuous so no log is flooded and no parallel monitoring surface appears:
a slow-poll line written only once a poll's classification work reaches
FM_SLOW_POLL_SECS, which is the loop-saturation threshold itself; and this
home's recorded endpoint count stamped onto the rare no-change heartbeat line,
so sustained concurrency has a historical series where previously only an
instantaneous reading existed. After the drain commits its authoritative output,
fm-wake-drain.sh appends how many distinct wakes the turn was handed and how long
the oldest waited - the delay between an event and the coordinator acting on it,
which nothing recorded at all. triage_log moves to fm-wake-lib.sh so the watcher
and the drain share one writer against one bound.
Deliberately not built: a structured supervision event bus with typed events and
schemas. One reader exists. A tag in prose costs fifteen lines; a bus costs a
schema, a version, and a second component that has to agree with the first.
Tests: branch tag rendered and recovered for tagged, detailed, both and bare
legacy forms; ledger classification per branch, including untagged and malformed
tags falling back to the classification they always had; branch assertions on
the wedge, pause-resurface, terminal, nonterminal and away-mode paths.
The wedge escalation asks whether a worker has stopped making progress, but its
only reset conditions were a pane-hash change and a busy signature - precisely
the two things a healthy worker driving a no-mistakes run cannot produce,
because the pipeline owns the branch and renders nothing to that worker's own
pane. A healthy validating worker therefore escalated on the same unchanged hash
every threshold, indefinitely: 28 of 61 stale wakes across one measured
22.5-hour window, on three panes, every inter-wake interval at or above
FM_STALE_ESCALATE_SECS plus one coordinator turn. Each one cost a coordinator
turn and each one concluded the worker was healthy.
Progress is a difference and nothing stored a previous value, so the health
verdict was structurally stateless. bin/fm-crew-state.sh --progress supplies that
value as a short fingerprint, the watcher stores it in state/.progress-<key>, and
wedge_timer_check compares the two.
The fingerprint's composition is the load-bearing part, and the obvious fields
are the wrong ones. Live `axi status` output carries active_for ("2h3m") and
last_activity ("quiet 1h25m ago: ...") on the active step; both embed a ticking
elapsed counter, so a fingerprint containing either differs on every read of an
unchanged run. That would reset the escalation unconditionally and leave a
genuinely frozen worker unable to escalate ever again - a false negative on the
one signal that catches a frozen worker, silent because nothing reports an
escalation that did not happen, and strictly worse than the false positive being
removed. agent_pid is empty on a live healthy step, so its presence proves
nothing; round is the non-numeric token "starting", so it is compared as an
opaque string. What remains is the monotonic completed-step count, the active
step's name and status, that opaque round token, and the worktree head - the head
covering scouts and pre-validation ship work, which have no attributed run and
for which a new commit is the only available evidence.
Three properties keep this from trading one defect for a worse one. The
comparison runs only on the escalation branch, so its one bounded call is spent
exactly where the alternative is a whole coordinator turn. With no stored
baseline it escalates as before, because absence of evidence is not evidence of
progress; what the fingerprint suppresses is the repeat escalations, each backed
by positive evidence that the run moved. An empty or unreadable fingerprint
compares equal and escalates.
Known and accepted narrowing, stated so it is not discovered later: a worker that
freezes while its pipeline keeps advancing is absorbed. That shape is real,
because the pipeline spawns its own agents. It is accepted because the
alternative today is escalating every healthy validating worker. If evidence ever
shows workers freezing under advancing pipelines, the fix is one more
deterministic term in the fingerprint, not a supervisor above this one.
Deliberately not built: a batch supervisor process, and model-mediated
supervision. Both exist to answer "is this worker healthy?", which is the exact
question this makes deterministic; making the existing supervisor
progress-aware removes the reason to add a tier above it.
Tests: two reads of an unchanged run whose ticking fields have both moved produce
a byte-identical fingerprint - the direct guard against those fields being
reintroduced, and it must not be dropped; the fingerprint moves on a completed
step, an active-step status change and a round change; with no attributed run it
is stable and moves on a new commit; a run without an active_steps table and a
torn-down worktree degrade to empty rather than to a false reset. End to end:
first round escalates without a baseline and records one, an advancing pipeline
absorbs and restarts the timer, a frozen pipeline re-escalates, and an unreadable
fingerprint escalates while preserving the stored baseline.
A declared pause on a crew whose agent is not confidently dead fails open to a surface, deliberately: such a crew may be sitting on a decision gate its own status line silenced. The defect is that this classification re-runs on every distinct pane hash, so the documented "looked at once" was implemented as "looked at once per pane redraw" - one wake per redraw, bypassing the re-surface throttle that lives only on the sibling dead-agent branch. The measured production rate was low, five wakes across one 22.5-hour window on a single live-agent pane. The reason to fix it is the ceiling rather than the rate: a pane that renders a ticking clock or a token counter produces a new hash every poll, so the same defect on such a task wakes the coordinator every 45 seconds indefinitely. Bounding an unbounded failure mode is worth forty lines even when the observed count is small. state/.paused-liveprobe-<key> records that this pause window has already had its one live-agent look. It is written where the surface actually happens, not where the classification decides, because that classification also returns the fail-open verdict on paths that do not surface and spending the budget there would consume the documented look without ever taking it. Once spent, a live agent takes the same bounded FM_PAUSE_RESURFACE_SECS cadence a dead one takes, so a forgotten pause still cannot rot invisibly. The marker is released when the crew is observed to no longer declare a pause - and that release is deliberately not gated on the sibling .paused-<key> file, which the hash-change path removes while the crew is still paused, so gating would strand the marker for the rest of the task's life. The first surface, the dead-agent behaviour, and the secondmate idle-endpoint exemption are all unchanged. This is only sufficient alongside park detection for an unresponsive gated run: throttling here can delay a real escalation by up to one re-surface window if a worker declares a pause and then hits a decision gate, and park detection is what closes that at ~120s regardless of pause state. Deliberately not built: hierarchical supervision. Hierarchy is warranted when one supervisor cannot keep up; one supervisor absorbs 95.9% of events in bash today. This removes an unbounded wake source without adding a component. docs/architecture.md changes in the same commit, from "first surfaces one stale wake" to "surfaces one stale wake per pause window". Tests: a live-agent declared pause across four pane redraws surfaces exactly once and records its look on the first; later redraws take the bounded cadence; leaving the pause releases the marker. The pre-existing dead-agent and captain-held cases are unchanged.
A crewmate received exactly one marked message in its life - the launch brief that fm-spawn wraps - and a bare stream of steers afterwards, indistinguishable from a human typing into its pane. The operating contract forbids a crewmate from addressing the captain, while the transport withheld the means to tell who was speaking. Both directions of that gap have been observed: a crewmate composed "Captain, the pipeline paused on one decision you need to make..." into its own pane and blocked for ten minutes on a reply that could never arrive, and the captain separately opened a crewmate pane believing it was firstmate. A metadata-routed steer to an ordinary crewmate or scout now carries the generic firstmate-steer kind from bin/fm-operational-input.sh - the carrier already exists, is already parsed, and already degrades gracefully for a worker that does not know it. Unlike the secondmate path it opens no reply expectation, because a steer expects no routed answer. Generated briefs teach the rule at the only moment a worker reads instructions. Two carve-outs. An explicit backend target and the key-send path stay bare, as before. And a message the harness itself must dispatch - a leading "/" slash command anywhere, or a leading "$" skill invocation on codex - is sent bare, because any prefix turns the command into plain text and the steer silently fails to run; "/no-mistakes" is the steer this protects. Those are the same two shapes fm-send's submit settle already recognizes as harness-dispatched, and the confusion being closed is about prose that reads like a person speaking, which a slash command cannot be mistaken for. Deliberately not built: extending the correlated pending-reply record from secondmates to crewmates. Acknowledgement protocols answer "sent but not received". The observed failure is a message never sent - the worker wrote prose into its own pane - which no acknowledgement protocol can detect, and building one would add correlation ids, a retry loop and a dead-letter path against zero observed instances of the failure it does catch. The existing asymmetry is correct: a secondmate round trip crosses homes with no shared filesystem convention for the reply, and a crewmate steer does not. Migration: crewmates spawned before this change receive marked steers their briefs never explained. The marker is a legible prefix, it degrades to visible text, and the property self-heals on the next dispatch. AGENTS.md and docs/architecture.md describe the marking boundary and change here. Tests: ship and scout selectors marked with firstmate-steer and never with the secondmate carrier; no pending-reply record created for a steer; slash commands and codex skill invocations stay bare while dollar-prefixed prose to a non-codex target is marked; explicit endpoints and the key path unchanged.
…espond A validation run that reaches a decision gate emits no wake of any kind. The gate state is computed correctly, but nothing polls for the transition - it is consulted only after a stale pane has already triggered classification, and a parked worker's pane may never go stale. A worker that does not answer its gate is therefore silent until something unrelated happens to look at it; the observed instance ran ten minutes and was caught only because firstmate inspected the pane during an unrelated wake. Zero instances appear in the logs, which is the defect rather than evidence against it: the event produces no log line at all. A bounded sweep reads at most FM_PARK_SCAN_MAX tasks every FM_PARK_SCAN_SECS and surfaces a gate only when it is seen unchanged across two sweeps. One sweep cannot tell an unanswered gate from one that is about to be answered promptly, and a run that is answered promptly must stay silent. Once surfaced, the same gate is not surfaced again until it changes. A ten-minute silence becomes a two-minute wake using the same authoritative state read supervision already makes, and fm-crew-state.sh stays the only owner of gate detection. This is the only wake class here that surfaces regardless of pane state, and the only increment in this series that ADDS wakes rather than removing them, so both bounds are configuration and FM_PARK_SCAN_MAX=0 switches it off entirely without a code edit. It sits with the other slow sweep and ahead of the per-wake paths for the same anti-starvation reason those are ordered that way: waking exits the cycle, so a sweep placed after them would be starved by a chatty sibling crewmate exactly when a quiet parked worker most needs noticing. It is also what makes the declared-pause throttle safe. Throttling a live-agent pause can delay a real escalation by up to one re-surface window if a worker declares a pause and then hits a gate; this closes that at roughly two minutes regardless of pause state. Known limit: only gates the run itself names are visible. A worker blocked on something outside its run is not, nothing detects that deterministically today, and detecting it would require the worker to declare it - which is the status protocol that already exists. Deliberately not built: model-mediated supervision, an LLM asked "is this worker stuck?". The park is a string comparison on a gate name. The genuinely interpretive question is already routed to firstmate by the deep-inspection demand, and routing it to a cheaper model yields a second-hand judgement firstmate has to re-derive - costing a model call and the coordinator turn. Tests: a first sighting stays silent and records the gate; the same gate on the next sweep surfaces once, naming it; an already-surfaced gate does not surface again; a run moving between gates stays silent while advancing its recorded gate; a run leaving its gate clears both markers so a later park starts its own count; and a zero cap disables the sweep without advancing any state.
The supervision fixes each declined a larger alternative, and the reasoning for declining is worth more to a future contributor than the outcome: without it the same proposals arrive again, are re-argued from scratch, and are adopted or rejected on taste rather than evidence. Each record names the problem it would solve, the evidence for and against it today, and a measurable trigger stated as a number against a named metric. A record without such a trigger is a graveyard entry rather than a dormant design, because nothing would ever cause anyone to look at it again; where a trigger is not collectable with what the system records, that is stated along with the instrument it needs. Two of these are captain decisions rather than implementation judgements, and both are recorded as decided rather than open. The batch abstraction is deferred, not rejected. The standard argument for defining an interface before it is exercised is that later migration is expensive; measured here it is close to free, because task metadata is a flat key-value file that has already absorbed four independent field additions from four writers with no migration, no version and no schema. The cost of building it now is concrete in the other direction: every semantic field in the plan-document schema is a guess, and the only fan-out-eligible work found in twenty real backlog items is four instances of one shape, so a schema written now would encode that shape as if it were general. The implementation stays shaped so the grouping key can be added when a real batch first exists. Speculative-parallel execution is dormant and structurally blocked rather than merely unbuilt. Every losing arm holds unlanded work, cleanup refuses to discard unlanded work without explicit captain authority, and that refusal exists because unlanded work has been lost before. An N-way round therefore costs N-1 captain authorizations forever, which increases the exact cost the mechanism would be adopted to reduce. Preserving compatibility for it would mean either the batch abstraction rejected above or relaxing that safety boundary, so no compatibility work is warranted. If the cleanup contract ever changes, the existing declared-scratch doctrine for scout worktrees supplies the discard mechanism - which is why generalizing that doctrine in advance is not needed. The file also records why instrumentation was preferred over a telemetry subsystem sized for hypothetical gate metrics, including the concrete reason continuous per-poll sampling was rejected: it would have written ~5,700 lines a day into the absorbed-wake log and evicted the very history that log exists for. Registered as maintainer-architecture and linked from architecture.md, which remains the owner of how supervision actually works. The progress fingerprint's field-level contract stays owned by bin/fm-crew-state.sh; this file carries only the architectural why-not and points at it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
Implement four approved supervision increments from an evidence-gated plan revision (data/supervision-plan-revision/report.md), each shipped as an independently reviewable fix: commit, under four explicit captain decisions.
GOAL: reduce false-positive supervision wakes measured at 61 stale wakes over 22.5 hours, without ever trading a false positive for a false negative. The captain was explicit: 'Preserving supervision correctness is significantly more important than minimizing implementation effort.'
THE FOUR CAPTAIN DECISIONS, which are architecture and not my choices to revisit:
DELIBERATE DECISIONS A REVIEWER READING ONLY THE DIFF WOULD LIKELY FLAG AS MISTAKES:
FIXES TO PRE-EXISTING DEFECTS FOUND DURING IMPLEMENTATION, included because they are prerequisites:
STRUCTURAL CHANGES THAT ARE MEANS, NOT ENDS:
TEST ASSERTIONS DELIBERATELY LOOSENED, which will look like weakening tests: fm-send-strict, fm-gate-refuse and fm-pending-reply asserted the exact literal typed text for a crewmate send. Those sends now carry the steer marker, so each was changed to assert the target plus the message BODY, with the marker itself pinned precisely in tests/fm-send-secondmate-marker.test.sh, which is its owner. This keeps one owner for the marker contract instead of duplicating it across four suites.
INSTRUMENTATION CHOICES, per decision 3: three measurements ride logs that already exist. Continuous per-poll duration sampling was REJECTED on measurement rather than taste: it would write ~5,700 lines a day into the absorbed-wake log and evict the history that log exists for, well inside its own size bound, destroying the evidence the instrument was added to produce. So the slow-poll line is written only once a poll reaches FM_SLOW_POLL_SECS, which is the saturation threshold itself, and fleet size rides the rare no-change heartbeat line. Supervision latency and drain depth were added because they express coordinator attention cost directly and nothing recorded them. A human-intervention rate and an average fan-out were deliberately NOT collected.
The new docs/supervision-dormant-designs.md is the required deliverable for captain decisions 1 and 2, plus the rationale for the three larger architectures each increment postpones. It deliberately does NOT restate the fingerprint's field-level contract, which stays owned by bin/fm-crew-state.sh's header; it carries only the architectural why-not and points at the code.
This repo is firstmate's own shared tracked material, so firstmate-coding-guidelines was loaded before editing: one owner per contract, cross-references rather than restatements, one sentence per line in tracked Markdown, plain dashes, no agent co-author, shellcheck-clean bin scripts, and colocated tests extending existing runners.
Validation already run locally: bin/fm-lint.sh clean, bin/fm-doc-audience-check.sh clean, and 15 touched suites green at 315 assertions. Six failures in the wider changed-selection run were each reproduced identically at base commit a5fe1bc and are pre-existing environment issues (three Node/Pi, one tmux smoke readiness, one session-start bootstrap diagnostic, one timing-sensitive watcher-lock restart-attach), not regressions from this branch.
What Changed
bin/fm-crew-state.sh --progressemits a short fingerprint (completed-step count, active step name and status, the opaqueroundtoken, worktree head — deliberately excluding the tickingactive_for/last_activityfields), the watcher stores it instate/.progress-<key>, andwedge_timer_checkabsorbs a repeat escalation only when the fingerprint proves the run moved; with no baseline, or an empty or degraded read, it escalates as before. A declared pause on a live agent now surfaces once per pause window viastate/.paused-liveprobe-<key>instead of once per pane redraw.bin/fm-watch.shthat wakes when a run sits unchanged at a decision gate across two sweeps — the one wake class this branch adds — bounded byFM_PARK_SCAN_SECS/FM_PARK_SCAN_MAXwith0disabling it, and rotated through a persistedstate/.park-scan-cursorso every ship task is covered withinceil(N/max)sweeps. Every stale reason now ends with a[branch=<name>]tag that the arm layer records asreason=actionable-stale-<branch>; window extraction moved to the single ownerstale_reason_window, fixingbin/fm-supervise-daemon.sh's bare${reason#stale: }strip, andtriage_logmoved tobin/fm-wake-lib.shso the watcher and wake drain share one bounded writer. Three sampled measurements ride existing logs: a slow-poll line atFM_SLOW_POLL_SECS, endpoint count on the heartbeat line, and drain depth plus oldest-wake age frombin/fm-wake-drain.sh.bin/fm-send.shnow marks metadata-routed crewmate and scout steers with the genericfirstmate-steeroperational-input kind, leaving slash commands, codex$skill invocations, explicit endpoints and the key-send path bare so harness-dispatched commands still run; generated briefs teach the rule.docs/supervision-dormant-designs.mdrecords the batch abstraction and speculative-parallel execution as decided-dormant with measurable revisit triggers, and the marker contract is pinned intests/fm-send-secondmate-marker.test.shwhile the four sibling suites assert target plus message body.Risk Assessment
✅ Low: Every finding from the prior round is fixed exactly as directed, both flagged hazards are respected (the liveprobe marker was added only to the daemon's cleanup, and the sha-only fingerprint form is preserved for genuine no-run cases), the new tests provably fail without their fixes, and only a minor non-functional duplication remains.
Testing
Ran the 15 touched suites (all green), then built four before/after harnesses that drive the real watcher, sender, crew-state helper and wake drain against both this branch and base commit a5fe1bc, capturing the coordinator-visible wake stream as CLI transcripts. Those transcripts show the headline goal met without the forbidden trade: a healthy validating worker drops from 6 wakes to 1 over six wedge thresholds while a genuinely frozen worker still escalates all 6; an unanswered decision gate behind a busy pane goes from silent to one wake naming the gate, disabled by FM_PARK_SCAN_MAX=0; a live-agent pause drops from 4 wakes to 1 across four redraws; crewmate and scout steers carry the firstmate-steer marker while slash commands, codex skill invocations and explicit endpoints stay bare; the --progress fingerprint is byte-identical when only the ticking fields move and empty when degraded; and the pre-existing daemon window-extraction defect is reproduced at base and fixed here. Artifacts are CLI transcripts rather than screenshots because this change has no rendered UI surface - the end-user experience is exactly the wake reason lines and pane keystrokes captured. Temp base checkout removed; worktree clean.
Evidence: Wedge escalation before/after: healthy worker 6 wakes -> 1, frozen worker 6 -> 6
SCENARIO 1 wedge escalation on a HEALTHY worker driving a no-mistakes run pane byte-identical throughout - the pipeline owns the branch BEFORE base commit a5fe1bc threshold 1 pipeline=3/ci/running/starting/9f1c2ab WAKE stale: captain:fm-ship-login (idle 501s, possible wedge, escalation 1) threshold 2 pipeline=4/ci/running/starting/9f1c2ab WAKE stale: captain:fm-ship-login (idle 500s, possible wedge, escalation 2) threshold 3 pipeline=5/ci/running/starting/9f1c2ab WAKE stale: ... escalation 3, demand-deep-inspection: ... threshold 4 pipeline=6/ci/running/starting/9f1c2ab WAKE stale: ... escalation 4, demand-deep-inspection: ... threshold 5 pipeline=7/ci/running/starting/9f1c2ab WAKE stale: ... escalation 5, demand-deep-inspection: ... threshold 6 pipeline=8/ci/running/starting/9f1c2ab WAKE stale: ... escalation 6, demand-deep-inspection: ... ==> 6 coordinator wakes over 6 wedge thresholds AFTER this branch threshold 1 pipeline=3/ci/running/starting/9f1c2ab WAKE stale: captain:fm-ship-login (idle 501s, possible wedge, escalation 1) [branch=wedge] threshold 2 pipeline=4/ci/running/starting/9f1c2ab absorbed - no coordinator turn threshold 3 pipeline=5/ci/running/starting/9f1c2ab absorbed - no coordinator turn threshold 4 pipeline=6/ci/running/starting/9f1c2ab absorbed - no coordinator turn threshold 5 pipeline=7/ci/running/starting/9f1c2ab absorbed - no coordinator turn threshold 6 pipeline=8/ci/running/starting/9f1c2ab absorbed - no coordinator turn ==> 1 coordinator wakes over 6 wedge thresholds SCENARIO 2 the same watcher on a GENUINELY FROZEN worker BEFORE base commit a5fe1bc ==> 6 coordinator wakes over 6 wedge thresholds AFTER this branch ==> 6 coordinator wakes over 6 wedge thresholds (every threshold still emits stale: ... possible wedge, escalation N [branch=wedge])Evidence: Park sweep: an unanswered gate behind a busy pane goes from silent to one named wake
SCENARIO 3 a run parked at a decision gate nobody answers pane stays BUSY the whole time, so no stale path can see it BEFORE base commit a5fe1bc t+120 s silent (gate recorded on disk: <none>) t+240 s silent (gate recorded on disk: <none>) t+360 s silent (gate recorded on disk: <none>) gate answered, run advances: silent (this task's park markers left on disk: 0) FM_PARK_SCAN_MAX=0 on a gate due to surface: silent - sweep disabled --- what the coordinator drains (fm-wake-drain.sh) --- AFTER this branch t+120 s silent (gate recorded on disk: parked at review: 2 finding(s)) t+240 s WAKE stale: captain:fm-ship-login (parked at review: 2 finding(s) across two sweeps - the run is waiting on a response the worker has not given) [branch=park] t+360 s silent (gate recorded on disk: parked at review: 2 finding(s)) gate answered, run advances: silent (this task's park markers left on disk: 0) FM_PARK_SCAN_MAX=0 on a gate due to surface: silent - sweep disabled --- what the coordinator drains (fm-wake-drain.sh) --- | 1785107633 1 stale captain:fm-ship-login stale: captain:fm-ship-login (parked at review: 2 finding(s) across two sweeps - the run is waiting on a response the worker has not given) [branch=park]Evidence: Declared pause on a live agent: 4 wakes across 4 pane redraws -> 1
SCENARIO 4 a declared pause on a LIVE agent, four pane redraws a pane with a ticking token counter rehashes every poll BEFORE base commit a5fe1bc pane redraw 1 WAKE stale: captain:fm-scout-api pane redraw 2 WAKE stale: captain:fm-scout-api pane redraw 3 WAKE stale: captain:fm-scout-api pane redraw 4 WAKE stale: captain:fm-scout-api ==> 4 coordinator wakes across 4 redraws of ONE pause window crew leaves the pause: live-probe marker released - the next pause gets its own look AFTER this branch pane redraw 1 WAKE stale: captain:fm-scout-api [branch=nonterminal] pane redraw 2 absorbed - this pause window already had its one look pane redraw 3 absorbed - this pause window already had its one look pane redraw 4 absorbed - this pause window already had its one look ==> 1 coordinator wakes across 4 redraws of ONE pause window crew leaves the pause: live-probe marker released - the next pause gets its own lookEvidence: fm-send: the literal text a crewmate, scout and harness command actually receive
AFTER this branch prose steer to a CREWMATE (kind=ship) typed by the operator : the captain wants the login fix rebased first lands in the pane as : <U+2063>FIRSTMATE_OP: v1 firstmate-steer: the captain wants the login fix rebased first prose steer to a SCOUT (kind=scout) typed by the operator : report what you found so far lands in the pane as : <U+2063>FIRSTMATE_OP: v1 firstmate-steer: report what you found so far SLASH COMMAND to a crewmate - must stay bare or it never dispatches typed by the operator : /no-mistakes lands in the pane as : /no-mistakes CODEX skill invocation - must stay bare on codex typed by the operator : $review the diff lands in the pane as : $review the diff dollar-prefixed PROSE to a non-codex crewmate - marked typed by the operator : $5 says this is worth doing lands in the pane as : <U+2063>FIRSTMATE_OP: v1 firstmate-steer: $5 says this is worth doing explicit backend endpoint - unchanged, stays bare typed by the operator : raw endpoint steer lands in the pane as : raw endpoint steer (BEFORE, at base a5fe1bc: every one of the six lands bare - indistinguishable from a human typing.)Evidence: --progress fingerprint beside the live axi status row it derives from
READ 1 - whatno-mistakes axi statusreports (the active_steps row): active_steps[1]{step,status,active_for,last_activity,agent_pid,round}: ci,running,1h22m,"quiet 43m56s ago: log: all CI checks passed - still monitoring until merged or closed","",starting fm-crew-state.sh ship-login --progress -> 3/ci/running/starting/a622a96 READ 2 - 41 minutes later, NOTHING about the run has advanced; only the two ticking fields moved (1h22m -> 2h3m, 43m56s -> 1h25m): ci,running,2h3m,"quiet 1h25m ago: log: all CI checks passed - still monitoring until merged or closed","",starting fm-crew-state.sh ship-login --progress -> 3/ci/running/starting/a622a96 => byte-identical. A frozen worker still escalates. READ 3 - the pipeline genuinely advances (ci running -> passed): fm-crew-state.sh ship-login --progress -> 3/ci/passed/starting/a622a96 => moved. The healthy worker's escalation is absorbed. READ 4 - the worktree is torn down / the run read fails (degraded): fm-crew-state.sh ship-login --progress -> [] => empty. An empty fingerprint compares equal, so the escalation still fires.Evidence: Pre-existing daemon defect: task id backed out of a decorated stale reason
BEFORE base commit a5fe1bc (bare ${reason#stale: } prefix strip) reason : stale: sess:fm-ship-login (idle 300s, possible wedge, escalation 2) [branch=wedge] window : sess:fm-ship-login (idle 300s, possible wedge, escalation 2) [branch=wedge] task : ship-login (idle 300s, possible wedge, escalation 2) [branch=wedge] reason : stale: sess:fm-ship-login (parked at review: 2 finding(s) across two sweeps) [branch=park] window : sess:fm-ship-login (parked at review: 2 finding(s) across two sweeps) [branch=park] task : 2 finding(s) across two sweeps) [branch=park] <-- garbage task id AFTER this branch (stale_reason_window, one owner) every one of the four forms -> window: sess:fm-ship-login , task: ship-loginEvidence: Gate instrumentation riding existing logs, and per-branch wake attribution
1. loop saturation - written only once a poll reaches FM_SLOW_POLL_SECS [2026-07-26T19:18:45-0400] slow poll: 1s of 1s over 1 recorded endpoint(s) 2. sustained concurrency - this home's endpoint count on the rare no-change heartbeat [2026-07-26T19:18:45-0400] absorbed heartbeat (no captain-relevant change) fleet=1 3. coordinator attention cost - depth and the oldest wake's wait, at drain time what the coordinator turn received: | 1785107746 1 stale captain:fm-ship-login stale: captain:fm-ship-login [branch=wedge] | 1785107891 1 stale captain:fm-scout-api stale: captain:fm-scout-api (parked at review: 2 finding(s)) [branch=park] [2026-07-26T19:18:53-0400] drain: depth=2 oldest_wait=187s 4. per-branch wake attribution in the existing cycle-exit ledger stale: ... [branch=wedge] -> reason=actionable-stale-wedge stale: ... [branch=park] -> reason=actionable-stale-park stale: ... [branch=pause-resurface] -> reason=actionable-stale-pause-resurface stale: ... [branch=nonterminal] -> reason=actionable-stale-nonterminal stale: captain:fm-ship-login -> reason=actionable-staleEvidence: A/B harness driving the real watcher at both commits (wedge, park, pause)
/tmp/no-mistakes-evidence/01KYG8NJXYY7ZMAR9SNRZJFVHN/demo-send-and-fingerprint.sh)Evidence: Harness for the three gate measurements and the ledger branch tags
Pipeline
Updates from git push no-mistakes
... (2 earlier update rounds omitted to keep the PR body within GitHub's 65536-char limit; full history is in the run log.)
bin/fm-crew-state.sh:298- The progress fingerprint does not distinguish "the run read failed" from "the run changed". crew_progress_fingerprint only populates the run-derived terms when HAVE_RUN=1 and RUN_SOURCE=full; when the bounded 10sno-mistakes axi statuscall (NM_TIMEOUT, fm-crew-state.sh:80) times out or returns empty, HAVE_RUN stays 0 and the fingerprint becomes////<head>-- non-empty, and different from the healthy4/ci/running/starting/abc1234. wedge_timer_check (bin/fm-watch.sh:343) tests only[ -n "$fp" ] && [ -n "$prev_fp" ] && [ "$fp" != "$prev_fp" ], so it reads that degradation as forward progress: it absorbs the escalation, restarts the timer and deletes the consecutive-escalation count. It absorbs a second time when the next read succeeds and the fingerprint flips back. The documented guarantee that "an empty or unreadable fingerprint compares equal and escalates" therefore holds only for a totally empty reader output, not for the partial read the timeout path actually produces. Fix by emitting a distinguishable no-evidence token when a run WAS expected but could not be read (KIND=ship with a CREW_BRANCH andno-mistakeson PATH, yet RUN_OUT empty or RUN_SOURCE=coarse) and having wedge_timer_check treat that token as compare-equal, so the reset stays keyed to positive evidence only. Note this must not swallow the legitimate scout / pre-validation case, where no run exists by design and the head sha is the only progress term.bin/fm-watch.sh:910- The park sweep can never see a ship window past the PARK_SCAN_MAX-th one. recorded_windows (bin/fm-watch.sh:261) iterates "$STATE"/*.meta in glob order, which is stable across sweeps, and the loop breaks once park_scanned reaches PARK_SCAN_MAX (default 3). There is no persisted cursor or rotation, so every sweep reads the same first three ship tasks and the fourth and later ones are never park-scanned at all. For a fleet larger than the cap this leaves the exact silent-failure hole the increment was added to close -- a run parked at a decision gate emitting no wake of any kind -- permanently open for those workers, while also re-spending all three bounded crew_parked_gate calls on the same tasks every interval. A persisted last-scanned window that the next sweep resumes after would keep both bounds intact while giving every window coverage within ceil(N/PARK_SCAN_MAX) sweeps.bin/fm-classify-lib.sh:414- Two different functions are named crew_progress_fingerprint with incompatible contracts: this one takes an <id> and shells out to "$FM_CREW_STATE_BIN" "$id" --progress, while bin/fm-crew-state.sh:296 takes no arguments and reads resolved globals. bin/fm-crew-state.sh sources fm-classify-lib.sh at its top, so its own definition only wins because it is declared later in the file; the# shellcheck disable=SC2119at bin/fm-crew-state.sh:591 exists solely to silence shellcheck noticing the collision. If the in-script definition is ever moved above the source line, renamed, or refactored into a sourced helper,fm-crew-state.sh <id> --progresswould call the classify-lib version and re-exec itself recursively. Rename the in-script one (e.g. compute_progress_fingerprint) so the two contracts have distinct names and the SC2119 suppression can be dropped.bin/fm-supervise-daemon.sh:468- The new .paused-liveprobe-<key> marker is absent from every explicit pause-tracking cleanup list: this daemon rm (which was extended for .progress- in the same commit), and clear_pause_state / clear_pause_tracking in bin/fm-watch.sh:401 and :404. Its only release paths are the watcher loop top (bin/fm-watch.sh:1006) and the else branch of surface_nonterminal_stale. When the daemon reconciles a window and clears .paused-, .paused-rechecked- and .paused-resurfaced- while the crew's last status line still reads paused or captain-held, the spent marker survives with nothing left to pair it with, and a later declared pause on that window is denied its one documented live-agent look until the watcher happens to observe a non-paused status. Adding it to the rm lists costs nothing and matches how the sibling markers are already handled.bin/fm-watch-arm.sh:305- The comment attributes the "[branch=<name>]" tag to "bin/fm-push-transition-lib.sh's stale_reason", but stale_reason is defined in bin/fm-classify-lib.sh:313; fm-push-transition-lib.sh is only one of its six callers. Under the repo's one-owner/cross-reference rule this points a reader at the wrong owner for the tag contract, which is the exact class of misdirection the branch tag was added to prevent.tests/fm-watch-triage.test.sh:1612- test_stale_reason_branch_tags sources bin/fm-watch-arm.sh into the suite shell, and it is invoked first in the runner list, so the effects apply to every subsequent test in the file. The arm script transitively sources bin/fm-wake-lib.sh, which with FM_HOME unset (the test harness never sets it) resolves FM_HOME and STATE to the repo root and runsmkdir -p <repo>/state, then rebinds the suite-wide STATE, TRIAGE_LOG, FM_WAKE_QUEUE and WATCH variables. The directory is gitignored and the ~45 later tests pass FM_STATE_OVERRIDE per subprocess so nothing breaks today, but the classifier only needs to be called three times: running it in a subshell (( . "$ROOT/bin/fm-watch-arm.sh"; watch_output_reason_type "$out" )) gets the same assertions without mutating the suite environment or writing into the working tree.🔧 Fix: fix(watch): rotate park sweep and empty degraded progress reads
1 info still open:
bin/fm-watch.sh:926- The park pre-pass calls window_to_task purely as a filter predicate and throws the result away, then the sweep loop recomputes it for each selected window (bin/fm-watch.sh:938). window_to_task greps every state/*.meta until it matches, so the pre-pass is now O(N) of those scans per sweep where the old hard break stopped after PARK_SCAN_MAX windows. This is not a regression worth blocking on -- the main stale loop already runs window_to_task per window on every poll, which is far more often than this sweep's PARK_SCAN_INTERVAL -- but the second call is pure duplication. Collecting the task into a parallel park_tasks array alongside park_windows, indexed identically, removes the recomputation and lets the now-dead[ -n "$task" ] || continueguard inside the loop go with it, since the pre-pass already filtered those windows out.✅ **Test** - passed
✅ No issues found.
bin/fm-test-run.sh tests/fm-crew-state.test.sh tests/fm-watch-triage.test.sh tests/fm-send-secondmate-marker.test.sh tests/fm-send-strict.test.sh tests/fm-gate-refuse.test.sh tests/fm-pending-reply.test.sh tests/fm-wake-queue.test.sh- 7 suites, all greenbin/fm-test-run.sh tests/fm-daemon.test.sh tests/fm-transition-lib.test.sh tests/fm-operational-input.test.sh tests/fm-brief.test.sh tests/fm-documentation-audiences.test.sh tests/fm-supervision-events.test.sh tests/fm-watch-checkpoint.test.sh tests/fm-instruction-owners.test.sh- 8 suites, all greenManual A/B harnessdemo-supervision-wakes.sh: drove the realbin/fm-watch.shfrom this branch and fromgit archive a5fe1bcthrough six consecutive wedge thresholds for a healthy (advancing) and a frozen pipeline, three park sweeps behind a busy pane, and four pane redraws of one declared pause - counting the wake reasons each watcher actually emittedManual harnessdemo-send-and-fingerprint.sh: ranbin/fm-send.sh(base and target) against stubbed tmux recordingsend-keys -l, printing the literal text landing in the pane for a crewmate steer, a scout steer,/no-mistakes, a codex$-skill, dollar-prefixed prose to a non-codex target, and an explicit endpointManual harnessdemo-send-and-fingerprint.sh: ranbin/fm-crew-state.sh ship-login --progressover a fixture reproducing the liveaxi statusactive_steps row verbatim, across two reads where only active_for/last_activity ticked, a real step-status change, and a degraded readManual harnessdemo-send-and-fingerprint.sh: exercisedstale_reason_window+window_to_taskon bare, wedge-tagged, pause-decorated and park-decorated stale reasons, against the base commit's bare${reason#stale: }stripManual harnessdemo-instrumentation.sh: ran a live watcher withFM_SLOW_POLL_SECS=0andFM_HEARTBEAT=2, thenbin/fm-wake-drain.shover a queue with known record ages, and classified five stale reasons throughbin/fm-watch-arm.sh'swatch_output_reason_typedocs/calm-mode-feasibility.md:135- The sentence "Current session-start, watcher, turn-end guard, away supervisor, and launch-brief inputs retain their versioned U+2063 static envelopes" enumerates five operational-input kinds; this change adds a sixth, firstmate-steer, to FM_OPERATIONAL_KINDS. I deliberately left it unedited: the page is classified maintainer-verification and records what was actually audited against Pi 0.81.1, so adding the new kind would claim coverage that was never verified. The authoritative kind list stays bin/fm-operational-input.sh. Worth a decision only if the calm-mode audit is re-run, which would also settle the related behavioural question of whether a Pi crewmate with calm on would render a firstmate-steer at all.✅ **Lint** - passed
✅ No issues found.
✅ **Push** - passed
✅ No issues found.