fix(watch): honor declared pause cadence for idle live panes

[thirtysevend]

Intent

Fix the firstmate watcher defect where an idle live pane whose latest status explicitly declares paused can emit repeated bare stale wakes as its rendered hash changes. A declared pause must absorb stale panes and re-surface only after FM_PAUSE_RESURFACE_SECS with the labeled declared-pause long-cadence reason; an authoritative active run must still override the pause as working, and any subsequent resolved or other new non-pause status must clear pause tracking and restore normal stale behavior. Preserve the stricter confirmed-dead rule for durable captain-held transfers and preserve AFK daemon handoff semantics. Add colocated watcher regression coverage for fresh pause absorption, changing-pane absorption, labeled cadence recheck, and unpause behavior, and update the authoritative architecture documentation.

What Changed

• pause_state_class in bin/fm-watch.sh now treats the latest explicit paused: status as authoritative while the endpoint idles: stale panes are absorbed and re-surfaced only every FM_PAUSE_RESURFACE_SECS with the labeled declared-pause long-cadence reason, instead of emitting repeated bare stale wakes as the rendered pane hash drifts. An authoritative active run still overrides the pause as working, and any later non-pause status (e.g. resolved:) clears pause tracking and restores normal stale handling. The stricter confirmed-dead requirement remains in place for durable captain-held transfers, and AFK daemon handoff semantics are unchanged.
• Rewrote the colocated watcher regression coverage in tests/fm-watch-triage.test.sh to exercise fresh pause absorption, changing-pane absorption, the labeled cadence recheck, and unpause behavior (full suite passes 47/47 in the pipeline Test gate).
• Updated docs/architecture.md to document the declared-pause authority, the active-run override, and the clear-on-new-status semantics.

Risk Assessment

✅ Low: A well-bounded single-function watcher fix that verifiably eliminates the repeated bare-wake path for declared pauses, preserves the adjacent captain-held and AFK semantics unchanged, satisfies every required intent criterion, and ships colocated regression coverage for all four required scenarios plus a matching documentation update.

Testing

Ran the colocated fm-watch-triage suite via the project's focused runner (all 47 tests pass, covering the new declared-pause regression tests plus the preserved captain-held confirmed-dead, active-run override, secondmate, and AFK handoff guarantees), then manually drove real fm-watch.sh subprocesses against the hermetic fixture on both the base and target commits: the base watcher reproduces the repeated bare-stale-wake defect, while the fixed watcher absorbs fresh and changing-pane declared pauses, re-surfaces once after FM_PAUSE_RESURFACE_SECS with the labeled long-cadence reason, and restores normal stale behavior on unpause. No visual artifact applies (headless CLI watcher; wake-queue transcripts are the end-user surface). Environment note: a broad pkill during test cleanup may have briefly killed the user's production watcher; it re-armed and is running.

Evidence: Before/after watcher demonstration transcript (defect on base vs fixed cadence)

== BEFORE (base fb368dc): live idle pane, latest status declares paused watcher re-arm #1: EXITED (surfaced a wake) wake-queue: stale | test:fm-gate | stale: test:fm-gate watcher re-arm #2 (pane hash drifted): EXITED (surfaced again) wake-queue: stale: test:fm-gate (x2) => defect: every re-arm of the declared-paused live pane appends another bare stale wake == AFTER (fix): same fixture, fresh declared pause absorbs watcher: ABSORBING (still armed after 5s, no exit); .paused marker present; wake-queue EMPTY == AFTER: pane content drifts mid-run - still absorbed watcher: ABSORBING; paused stale suppressor advanced; wake-queue EMPTY == AFTER: pause older than FM_PAUSE_RESURFACE_SECS (240s) watcher: EXITED once for the scheduled recheck wake-queue: stale: test:fm-gate (paused 501s, awaiting external - declared pause, rechecked on a long cadence not a wedge; confirm the wait still holds) == AFTER: crew writes 'resolved:' - pause clears watcher: EXITED with a normal stale wake (stale: test:fm-gate); pause tracking marker: cleared

== BEFORE (base fb368dcc6380bfba5b4ba35722106692f3e789b3): live idle pane, latest status declares paused
   watcher re-arm #1: EXITED (surfaced a wake)
   wake-queue (kind | key | payload):
     stale | test:fm-gate | stale: test:fm-gate
   watcher re-arm #2 (pane hash drifted): EXITED (surfaced again)
   wake-queue (kind | key | payload):
     stale | test:fm-gate | stale: test:fm-gate
     stale | test:fm-gate | stale: test:fm-gate
   => defect: every re-arm of the declared-paused live pane appends another bare stale wake

== AFTER (fix under test): same fixture, fresh declared pause absorbs
   watcher: ABSORBING (still armed after 5s, no exit)
   pause tracking marker .paused-test_fm-gate: present
   wake-queue: EMPTY

== AFTER: pane content drifts mid-run - still absorbed
   watcher: ABSORBING (alive across the pane change)
   paused stale suppressor advanced to the drifted pane hash
   wake-queue: EMPTY

== AFTER: pause older than FM_PAUSE_RESURFACE_SECS (240s) - labeled long-cadence recheck
   watcher: EXITED once for the scheduled recheck
   wake-queue (kind | key | payload):
     stale | test:fm-gate | stale: test:fm-gate (paused 501s, awaiting external - declared pause, rechecked on a long cadence not a wedge; confirm the wait still holds)

== AFTER: crew writes 'resolved:' - pause clears, normal stale behavior returns
   watcher: EXITED with a normal stale wake
   wake-queue (kind | key | payload):
     stale | test:fm-gate | stale: test:fm-gate
   pause tracking marker: cleared

Evidence: Colocated fm-watch-triage suite run (47/47 ok via bin/fm-test-run.sh)

FM_TEST_BEGIN 2026-08-07T00:46:13Z tests/fm-watch-triage.test.sh family=watcher-wake-lock expected_gate_skip=none
ok - signal_reason_is_actionable: benign absorbed, captain verbs and coalesced batches surfaced
ok - stale_is_terminal: terminal status surfaces, non-terminal and no-status are benign
ok - scan_captain_relevant_statuses lists only captain-relevant statuses
ok - classifier primitives: keyed decisions and activity phases, captain relevance, window-to-task, and overrides
ok - crew_is_provably_working: only working+run-step/pane is provable; idle/finished/parked/failed/unknown surface
ok - status_is_paused: only the leading paused verb matches, and paused is not captain-relevant
ok - crew_absorb_class: working/paused/none from one read; crew_is_paused and crew_is_provably_working agree
ok - signal_crew_provably_working: benign only when every referenced crew is provably working
ok - a no-verb signal whose crew is provably working is absorbed (no exit, no queue, suppressor advanced, beacon present)
ok - a bare turn-end whose crew is provably working (busy pane) is absorbed
ok - a bare turn-end whose crew is not provably working is surfaced (the swallowed-finish fix)
ok - a no-verb working: note whose crew is idle with no running pipeline is surfaced
ok - captain-relevant signal is surfaced (queue + exit) and marked surfaced
ok - a stale pane sitting on a terminal status is surfaced (queue + exit)
ok - a stale terminal-looking status is overridden and absorbed while a run is actively working, then wedge-escalated
ok - provably-working non-terminal stale is absorbed on first sight, then wedge-escalated past the threshold
ok - consecutive wedge escalations on the same pane accumulate and demand deep inspection at the threshold
ok - a pane becoming active again resets the consecutive wedge-escalation counter
ok - a busy worker below the turn-age bound remains working with no escalation
ok - a busy worker with a stable pane hash still escalates once its completed-turn age reaches the bound
ok - a busy worker whose pane hash changes every poll still escalates once its completed-turn age reaches the bound
ok - touching a busy worker's completed-turn marker resets the age and prevents an old-age escalation
ok - repeated busy turn-age escalations reuse the existing escalation counter and demand deep inspection at the threshold
ok - the production default busy-turn-age bound is 3600s (5min under does not wedge, 66min over does)
ok - a not-provably-working non-terminal stale is surfaced immediately (never left to wait out the timer)
ok - a declared pause is absorbed on first sight, then re-surfaced as a recheck past the threshold, never wedge-escalated
ok - declared pauses use bounded labeled cadence and a later status resumes normal stale behavior
ok - a declared paused secondmate re-surfaces on the bounded normal-mode cadence
ok - a non-paused secondmate retains normal stale suppression
ok - a resumed secondmate clears pause and stale tracking before stale exemption
ok - unchanged stale hashes reclassify when a crew enters or leaves pause
ok - a declared pause is periodically rechecked against authoritative active-run state
ok - a paused status overridden by authoritative working preserves its wedge timer and escalates
ok - matching non-terminal stale suppressors repair missing or corrupt stale-since timers
ok - triage log capping handles wc byte counts with leading spaces
ok - a captured process-event result wakes a healthy watcher proactively, with no manual drain
ok - a process-event wake is delivered once: no duplicate wake while queued, and none once handled
ok - complete process-event queue keys map to distinct seen markers
ok - queue revalidation, proactive output, and marker commit serialize with drain
/srv/agents/users/firstmate/.no-mistakes/worktrees/d45d171c4de4/01KZCTJERFSJ2AGXM9TWAQZQWQ/bin/fm-push-transition-lib.sh: line 92: echo: write error: Broken pipe
tests/wake-helpers.sh: line 254: 3838256 Killed                  PATH="$dir/fakebin:$PATH" FM_HOME="$dir" FM_PROCEVENT_CLAIM_ROOT="$dir/claims" FM_CREW_STATE_BIN="$dir/fakebin/fm-crew-state.sh" FM_POLL=0.2 FM_SIGNAL_GRACE=1 FM_CHECK_INTERVAL=999999 FM_HEARTBEAT=999999 "$WATCH" > "$out"
tests/wake-helpers.sh: line 254: 3838823 Killed                  PATH="$dir/fakebin:$PATH" FM_HOME="$dir" FM_PROCEVENT_CLAIM_ROOT="$dir/claims" FM_CREW_STATE_BIN="$dir/fakebin/fm-crew-state.sh" FM_POLL=0.2 FM_SIGNAL_GRACE=1 FM_CHECK_INTERVAL=999999 FM_HEARTBEAT=999999 "$WATCH" > "$out"
ok - surfacing failures replay before marker commit and suppress only after delivered output
ok - marker failure exits through the shared wake owner, releases its lock, and replays later
ok - a heartbeat with no captain-relevant change is absorbed and backs off the cadence
ok - heartbeat backstop fail-safe surfaces a captain-relevant status the per-wake path missed
ok - the liveness beacon stays fresh while the watcher absorbs benign wakes (fm-guard never false-alarms)
ok - with .afk present the watcher reverts to one-shot so the daemon owns triage (no double-triage)
ok - AFK changed paused panes hand off plain stale identities for daemon-owned pause triage
FM_TEST_END 2026-08-07T00:48:27Z tests/fm-watch-triage.test.sh exit=0 duration_ms=133824 gate_skip=false
FM_TEST_SUMMARY total=1 failed=0 skipped_gate=0 duration_ms=133868
FM_TEST_SUMMARY_FAMILY family=watcher-wake-lock count=1 duration_ms=133824 failed=0
FM_TEST_SLOWEST rank=1 script=tests/fm-watch-triage.test.sh duration_ms=133824

Evidence: Manual demonstration script (reproducible before/after steps)

#!/usr/bin/env bash
# Manual end-to-end demonstration for the declared-pause cadence fix.
# Drives a REAL fm-watch.sh subprocess against the hermetic test fixture
# (fake tmux + fake fm-crew-state) in two configurations:
#   BASE  = bin/fm-watch.sh from fb368dc (before the fix)
#   FIXED = bin/fm-watch.sh from the change under test (703a6dd)
# Scenario: an idle LIVE pane whose latest status explicitly declares
# "paused [key=route]: waiting on an external dependency", with rendered pane
# content that drifts between watcher re-arms.
set -u
WT=${WT:?worktree path}
BASE_SHA=fb368dcc6380bfba5b4ba35722106692f3e789b3

# shellcheck source=/dev/null
. "$WT/tests/wake-helpers.sh"
TMP_ROOT=$(fm_test_tmproot fm-pause-demo)

# Extract the pre-fix toolbelt so its watcher resolves its own-era libraries.
BASEBIN_ROOT="$TMP_ROOT/base-checkout"
mkdir -p "$BASEBIN_ROOT"
git -C "$WT" archive "$BASE_SHA" bin | tar -x -C "$BASEBIN_ROOT"

WINDOW="test:fm-gate"
KEY=$(printf '%s' "$WINDOW" | tr ':/.' '___')
seen_sig() { stat -c '%s:%Y' "$1" 2>/dev/null; }
reap() { kill "$1" 2>/dev/null || true; wait "$1" 2>/dev/null || true; }

say() { printf '\n== %s\n' "$*"; }
show_queue() {
  local state=$1
  if [ -s "$state/.wake-queue" ]; then
    printf '   wake-queue (kind | key | payload):\n'
    awk -F '\t' '{ printf "     %s | %s | %s\n", $3, $4, $5 }' "$state/.wake-queue"
  else
    printf '   wake-queue: EMPTY\n'
  fi
}

new_case() {  # <name> -> echoes case dir
  local dir
  dir=$(make_case "$1")
  printf 'window=%s\nkind=ship\nharness=grok\nbackend=tmux\n' "$WINDOW" > "$dir/state/gate.meta"
  printf 'paused [key=route]: waiting on an external dependency\n' > "$dir/state/gate.status"
  printf '%s' "$(seen_sig "$dir/state/gate.status")" > "$dir/state/.seen-gate_status"
  printf '%s\n' "$dir"
}

seed_pane() {  # <case-dir> <pane text>
  local dir=$1 text=$2
  printf '%s\n' "$text" > "$dir/pane.txt"
  printf '%s' "$(hash_text "$text")" > "$dir/state/.hash-$KEY"
  printf '1\n' > "$dir/state/.count-$KEY"
}

arm() {  # <watch-bin> <case-dir> <resurface-secs> <crew-verdict> -> pid
  local watch=$1 dir=$2 resurface=$3 verdict=$4
  PATH="$dir/fakebin:$PATH" FM_FAKE_TMUX_WINDOW="$WINDOW" FM_FAKE_TMUX_CAPTURE="$dir/pane.txt" \
    FM_FAKE_TMUX_CURRENT_COMMAND=grok FM_FAKE_CREW_STATE="$verdict" \
    FM_STATE_OVERRIDE="$dir/state" FM_CREW_STATE_BIN="$dir/fakebin/fm-crew-state.sh" \
    FM_PAUSE_RESURFACE_SECS="$resurface" FM_POLL=1 FM_SIGNAL_GRACE=1 \
    FM_CHECK_INTERVAL=999999 FM_HEARTBEAT=999999 "$watch" >> "$dir/watch.out" &
  printf '%s\n' "$!"
}

wait_exit() {  # <pid> <secs>
  local pid=$1 secs=$2 i=0
  while [ "$i" -lt $((secs * 10)) ]; do
    kill -0 "$pid" 2>/dev/null || return 0
    sleep 0.1; i=$((i + 1))
  done
  return 1
}

PAUSED_VERDICT='state: paused · source: status-log · waiting on an external dependency'

# ---------------------------------------------------------------------------
say "BEFORE (base $BASE_SHA): live idle pane, latest status declares paused"
dir=$(new_case before)
seed_pane "$dir" 'idle external-decision gate 00:01'
pid=$(arm "$BASEBIN_ROOT/bin/fm-watch.sh" "$dir" 999 "$PAUSED_VERDICT")
if wait_exit "$pid" 20; then
  printf '   watcher re-arm #1: EXITED (surfaced a wake)\n'
else
  reap "$pid"; printf '   watcher re-arm #1: still absorbing\n'
fi
show_queue "$dir/state"

# firstmate re-arms the watcher; the idle pane's rendered content drifted.
seed_pane "$dir" 'idle external-decision gate 00:02'
pid=$(arm "$BASEBIN_ROOT/bin/fm-watch.sh" "$dir" 999 "$PAUSED_VERDICT")
if wait_exit "$pid" 20; then
  printf '   watcher re-arm #2 (pane hash drifted): EXITED (surfaced again)\n'
else
  reap "$pid"; printf '   watcher re-arm #2 (pane hash drifted): still absorbing\n'
fi
show_queue "$dir/state"
printf '   => defect: every re-arm of the declared-paused live pane appends another bare stale wake\n'

# ---------------------------------------------------------------------------
say "AFTER (fix under test): same fixture, fresh declared pause absorbs"
dir=$(new_case after)
seed_pane "$dir" 'idle external-decision gate 00:01'
pid=$(arm "$WT/bin/fm-watch.sh" "$dir" 999 "$PAUSED_VERDICT")
if wait_exit "$pid" 5; then
  printf '   watcher: EXITED unexpectedly\n'; cat "$dir/watch.out"
else
  printf '   watcher: ABSORBING (still armed after 5s, no exit)\n'
fi
[ -e "$dir/state/.paused-$KEY" ] && printf '   pause tracking marker .paused-%s: present\n' "$KEY"
show_queue "$dir/state"

say "AFTER: pane content drifts mid-run - still absorbed"
printf 'idle external-decision gate 00:02\n' > "$dir/pane.txt"
new_hash=$(hash_text 'idle external-decision gate 00:02')
i=0
while [ "$i" -lt 100 ] && kill -0 "$pid" 2>/dev/null; do
  [ "$(cat "$dir/state/.stale-$KEY" 2>/dev/null || true)" = "$new_hash" ] && break
  sleep 0.1; i=$((i + 1))
done
if kill -0 "$pid" 2>/dev/null; then
  printf '   watcher: ABSORBING (alive across the pane change)\n'
else
  printf '   watcher: EXITED unexpectedly\n'; cat "$dir/watch.out"
fi
[ "$(cat "$dir/state/.stale-$KEY" 2>/dev/null || true)" = "$new_hash" ] \
  && printf '   paused stale suppressor advanced to the drifted pane hash\n'
show_queue "$dir/state"
reap "$pid"

say "AFTER: pause older than FM_PAUSE_RESURFACE_SECS (240s) - labeled long-cadence recheck"
back=$(( $(date +%s) - 500 ))
touch -m -d "@$back" "$dir/state/gate.status"
printf '%s' "$(seen_sig "$dir/state/gate.status")" > "$dir/state/.seen-gate_status"
printf 'idle external-decision gate 00:09\n' > "$dir/pane.txt"
pid=$(arm "$WT/bin/fm-watch.sh" "$dir" 240 "$PAUSED_VERDICT")
if wait_exit "$pid" 20; then
  printf '   watcher: EXITED once for the scheduled recheck\n'
else
  reap "$pid"; printf '   watcher: did NOT re-surface\n'
fi
show_queue "$dir/state"

say "AFTER: crew writes 'resolved:' - pause clears, normal stale behavior returns"
printf 'resolved [key=route]: external dependency cleared\n' >> "$dir/state/gate.status"
printf '%s' "$(seen_sig "$dir/state/gate.status")" > "$dir/state/.seen-gate_status"
printf 'idle after external dependency cleared\n' > "$dir/pane.txt"
: > "$dir/state/.wake-queue"
: > "$dir/watch.out"
pid=$(arm "$WT/bin/fm-watch.sh" "$dir" 999 'state: stopped · source: pane · idle after dependency cleared')
if wait_exit "$pid" 20; then
  printf '   watcher: EXITED with a normal stale wake\n'
else
  reap "$pid"; printf '   watcher: did NOT surface\n'
fi
show_queue "$dir/state"
if [ -e "$dir/state/.paused-$KEY" ]; then
  printf '   pause tracking marker: STILL PRESENT (bad)\n'
else
  printf '   pause tracking marker: cleared\n'
fi

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

⚠️ **Review** - 1 info

• ℹ️ bin/fm-watch.sh:380 - In pause_state_class's new declared-pause branch (bin/fm-watch.sh:380), while .paused-<key> exists and the recheck marker is fresher than STALE_ESCALATE_SECS, the function returns 'paused' without consulting crew_absorb_class, so an authoritative active run flips the classification to 'working' only after the recheck window (default 240s) expires. This is bounded, the pane is absorbed either way during the window, and it mirrors the pre-existing captain-held throttle design — noting it only as an acknowledged eventual-consistency tradeoff against the intent's 'active run must override' criterion.

✅ **Test** - passed

✅ No issues found.

• bin/fm-test-run.sh tests/fm-watch-triage.test.sh — full colocated watcher-triage suite, 47/47 ok, exit 0, including the rewritten test_declared_pause_and_exited_captain_hold_use_bounded_cadence (fresh pause absorption, changing-pane absorption, labeled cadence recheck, unpause) and the preserved-guarantee tests (test_paused_authoritative_working_preserves_wedge_timer, test_nonterminal_paused_rechecks_authoritative_state, test_afk_present_reverts_watcher_to_one_shot, test_afk_paused_changed_pane_hands_off_plain_stale, secondmate pause/unpause tests)
• Manual end-to-end before/after demonstration (declared-pause-demo.sh): drove the real fm-watch.sh from the base commit fb368dc against a live declared-paused fixture pane — each of two watcher re-arms with a drifted pane hash appended another bare stale: test:fm-gate wake (defect reproduced)
• Same manual fixture against the fixed watcher: fresh declared pause absorbed (watcher stays armed, pause marker recorded, wake queue empty), pane-content drift absorbed with the paused stale suppressor advanced, a 500s-old pause re-surfaced exactly once with the labeled declared pause, rechecked on a long cadence reason, and a subsequent resolved: status cleared pause tracking and emitted the normal bare stale wake
• Verified docs/architecture.md in the diff documents the new declared-pause authority, active-run override, and unpause semantics

✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.