diff --git a/AGENTS.md b/AGENTS.md index c1e5272828..5d65d7db97 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -72,6 +72,7 @@ config/secondmate-harness harness the PRIMARY uses to launch SECONDMATE agents, config/backlog-backend backlog backend override; LOCAL, gitignored; absent or "tasks-axi" = default tasks-axi backend, "manual" = force routine backlog updates to hand-editing; inherited by secondmate homes (section 10) config/backend runtime session-provider backend override for new tasks; LOCAL, gitignored; absent = falls through to runtime auto-detection (the runtime firstmate itself is executing inside), then tmux; tmux is the verified reference backend (docs/tmux-backend.md), while herdr, zellij, orca, and cmux are experimental spawn backends (docs/herdr-backend.md, docs/zellij-backend.md, docs/orca-backend.md, docs/cmux-backend.md) - herdr and cmux can also be selected by runtime auto-detection, zellij and orca never are (always explicit), and codex-app is not accepted; see docs/codex-app-backend.md; inherited by secondmate homes under the primary-authoritative contract in secondmate-provisioning config/calm Pi Calm presentation preference; LOCAL, gitignored, and not inherited; see docs/configuration.md "Pi Calm preference" +config/concurrency-floor optional target of concurrent live ships per home; LOCAL, gitignored, absent or 0 = off; see docs/configuration.md "Ship concurrency floor" config/startup-memory-budget primary-authoritative per-home startup-memory budget; LOCAL, gitignored, materialized as 7,500 estimated tokens by locked primary bootstrap and inherited into secondmate homes; see docs/configuration.md "Startup memory budget" config/herdr-presentation-spaces optional presence flag for Herdr's default-off disposable single-task visual projection; LOCAL, gitignored; inherited by secondmate homes; see docs/herdr-backend.md "Optional presentation spaces" config/cmux-socket-password optional cmux control-socket password; LOCAL, gitignored; read fresh on every cmux CLI call and passed through without ever overriding an operator's own ambient CMUX_SOCKET_PASSWORD when absent (docs/cmux-backend.md "Setup") @@ -110,6 +111,8 @@ state/ volatile runtime signals; gitignored public-followup/ generated private transport for promised public replies: commitment registrations, typed terminal-result inbox, accepted/rejected ledgers (section 14; bin/fm-public-followup.sh) x-poll.error x-poll.claim-error generated X-mode relay and offer-claim diagnostic dedupe markers .wake-queue durable queued wakes: epochseqkindkeypayload + .refill-completion- crash-recoverable idempotence receipt for one completion-refill wake per task (bin/fm-refill-lib.sh) + .refill-needed-completion .refill-needed-floor durable home-level refill supervision markers (bin/fm-refill-lib.sh) .afk durable away-mode flag; present = sub-supervisor may inject escalations (set by /afk, cleared on user return) .watch.lock .wake-queue.lock watcher singleton and queue serialization locks .claude-autoarm.lock .claude-autoarm-epoch .turnend-claude-blocks Claude Stop auto-arm single-flight, epoch, and guard-budget records; never touch @@ -334,6 +337,7 @@ Tear down a ship task only after landing is confirmed. A teardown refusal for uncommitted or unlanded work is a stop-and-investigate result, never an obstacle to bypass. Never force teardown without explicit discard authority. After successful teardown, record completion, retain only the configured recent Done history, and re-evaluate queued work whose blockers and time gates have cleared. +Teardown and merge-poll completion also emit a durable refill wake (`bin/fm-refill-lib.sh`); treat that wake as the claim-next trigger rather than relying on mid-turn memory of a printed reminder. A secondmate is persistent and an empty queue is healthy. Retire one only on an explicit captain or main-firstmate decision, after loading `secondmate-provisioning`; its home must contain no work under way, and forced discard still requires explicit captain authority. @@ -350,11 +354,11 @@ The promoted worker must inventory scratch state, return to a clean default-bran Fleet supervision is an always-loaded operational contract; `docs/architecture.md`, `docs/turnend-guard.md`, the emitted session-start block, and script help own mechanisms and harness-specific recipes. -Whenever work is under way, keep exactly one live supervision cycle using the emitted protocol for this primary harness. -X mode may require that same live cycle with no fleet work. +Whenever work is under way or refill remains pending, keep exactly one live supervision cycle using the emitted protocol for this primary harness. +X mode or an unmet concurrency floor may require that same live cycle with no fleet metadata. Do not substitute another harness's wait shape, use shell `&`, or create a second cycle when a healthy one already exists. For every actionable wake, follow the ordinary-wake continuation in the emitted protocol; use its repair action only when the live cycle is missing or failed. -No turn ends blind while work is under way, including turns described as holding or waiting. +No turn ends blind while supervision remains needed, including work described as holding or waiting. At the start of every wake-handling turn, drain the durable wake queue before peeking, reading beyond the reason line, steering, or starting work. Session start is the only exception because its one-shot digest already drained while locked or deliberately left the queue untouched in lock-refused read-only mode. @@ -365,7 +369,10 @@ Handle actionable wakes as follows: 1. For `signal:`, read the listed event lines first, then reconcile current state only where action depends on it. 2. For `stale:`, inspect the recorded endpoint and load `stuck-crewmate-recovery` for a stopped, looping, confused, or unresponsive worker; a deep-inspection reason also requires current-state and validation-log inspection. -3. For `check:`, act on the named poll result, including merges and X-mode events. +3. For `check:`, act on the named poll result, including merges, X-mode events, and refill signals. + A `check: refill completion :`, `check: refill floor:`, or `check: refill pending:` wake is claim-next capacity, not a blind spawn: run the normal claim-and-dispatch procedure (verify-at-pickup, atomic claim, exclusions, held and parked items, date gates). + A successful ship spawn clears the handled completion need automatically; after a completed attempt that finds no ready work, no eligible work, or held-only work, run `bin/fm-refill-complete.sh` with the matching `no-ready`, `no-eligible`, or `held-only` outcome. + Until ship worktree leases land, hold pool dispatch unless `fm_refill_has_parked_unpushed` proves every parked ship worktree readable and free of unpushed commits (`bin/fm-refill-lib.sh`). 4. For `heartbeat:`, review the whole fleet from the structured fleet view, reconcile suspicious tasks and PR state, update the backlog, and never report an unchanged fleet as progress. When any wake reports a merged PR for a project cloned in this home, refresh that clone through the guarded fleet-sync path. diff --git a/README.md b/README.md index 84ec32f3a9..c2197f7df6 100644 --- a/README.md +++ b/README.md @@ -47,7 +47,7 @@ Launching a supported harness inside it instantiates your first mate - and makes - **Two task shapes** - ship tasks deliver authorized changes; scout tasks leave standalone investigation reports when the intake contract warrants separate research. - **Explicit project modes** - each project ships via `no-mistakes`, `direct-PR`, or `local-only`, with an optional `+yolo` autonomy flag. - **Optional secondmates** - opt in to persistent second mates that run from isolated firstmate homes with their own `FM_HOME`, state, projects, and session lock, supervising project clones or a project-less firstmate-repo domain, kept on the primary firstmate version by guarded local fast-forwards and checked for live agent processes at session start. -- **Event-driven, zero-token supervision** - a bash watcher sleeps on the fleet and wakes the first mate only when something needs you; verified primary harnesses also get a turn-end backstop that blocks or follows up on a blind stop when work is under way and supervision is not live. +- **Event-driven, zero-token supervision** - a bash watcher sleeps on the fleet and wakes the first mate only when something needs you; verified primary harnesses also get a turn-end backstop that blocks or follows up on a blind stop whenever fleet, relay, or refill work needs supervision and no watcher is live. - **Optional X mode** - opt in with one local `.env` token so firstmate can answer your public `@myfirstmate` mentions, act on normal reversible mention requests through the same lifecycle as chat requests, acknowledge spawned work, and post up to three public-safe completion follow-ups within seven days for genuine milestones and the final outcome without changing non-X behavior; a final reply promised in a thread becomes durable state that is reconciled from disk, so a restart or a compacted conversation cannot lose it; dry-run preview records would-be replies and dismissals locally before go-live. - **Strict project boundary** - the first mate is read-only over your projects except for the narrow guarded and captain-approved operations authorized by [hard rule 1](AGENTS.md#1-identity-and-prime-directives), including fleet sync's guarded safe branch pruning; crewmates make every other project change behind the configured merge authority. - **Restart-proof** - all state lives on disk and in the active session backend (tmux by hard default, herdr or cmux when selected or auto-detected, zellij/orca when explicitly selected); kill the session anytime and the next one reconciles, including confirmed-dead secondmate agents, and carries on. diff --git a/bin/fm-claude-stop-autoarm.sh b/bin/fm-claude-stop-autoarm.sh index df9ee1128f..26e40d7f02 100755 --- a/bin/fm-claude-stop-autoarm.sh +++ b/bin/fm-claude-stop-autoarm.sh @@ -18,8 +18,8 @@ # - AFK: while state/.afk exists the away daemon owns the watcher and triage; # this hook exits 0 and NEVER rewakes the primary (checked again at # translation time so a mid-cycle AFK transition is honored). -# - Need: arms only while work is in flight (state/*.meta) or X mode has a -# relay poll to run (state/x-watch.check.sh); an idle home exits 0. +# - Need: arms while work is in flight, X mode has a relay poll, or refill +# work remains queued, pending, or below a configured floor. # - Single-flight: Claude does not dedupe async hooks, so a home-scoped owner # lock (state/.claude-autoarm.lock) admits exactly one owner; every other # concurrent firing exits 0 without translating, which keeps one event @@ -89,7 +89,7 @@ fi # --- AFK: the away daemon owns the watcher and triage; never rewake ---------- [ -e "$STATE/.afk" ] && exit 0 -# --- need: in-flight work or an X-mode relay poll ---------------------------- +# --- need: in-flight work, X-mode relay polling, or refill work -------------- need_supervision() { fm_supervision_needed "$STATE" "$GRACE" } diff --git a/bin/fm-guard.sh b/bin/fm-guard.sh index e36b7f46b0..205666ac77 100755 --- a/bin/fm-guard.sh +++ b/bin/fm-guard.sh @@ -5,7 +5,7 @@ # First, always warn if the firstmate primary checkout (FM_ROOT) is on a named # non-default branch, because that means firstmate-on-itself work landed in the # primary instead of an isolated worktree. -# Then, if any task is in flight (a state/.meta exists) and the watcher's +# Then, if any task, X-mode relay, or refill work needs supervision and the watcher's # liveness beacon (state/.last-watcher-beat, touched every poll cycle) is # missing or older than FM_GUARD_GRACE seconds, prints a loud, clearly delimited # banner so the agent cannot skim past it in the tool output of whatever it was @@ -140,27 +140,26 @@ if [ -n "$tangle_branch" ]; then } >&2 fi -# Compute in-flight count and watcher-beacon freshness via the shared -# grace-based predicate (bin/fm-supervision-lib.sh). Only act with tasks in -# flight; count them so the banner can say how much is riding on an absent -# watcher. +# Compute fleet, relay, and refill supervision need plus watcher-beacon freshness +# through the shared grace-based predicate (bin/fm-supervision-lib.sh). Keep the +# in-flight count so the banner can say how much is riding on an absent watcher. fm_supervision_status "$STATE" "$GRACE" in_flight=$FM_SUP_IN_FLIGHT watcher_fresh=$FM_SUP_WATCHER_FRESH beacon_desc=$FM_SUP_BEACON_DESC -if [ "$in_flight" -eq 0 ]; then - # Leave the unhealthy state (no work riding on the watcher): clear so a later - # in-flight + stale combination is a fresh episode even if the beacon is still - # absent with the same key string. +if [ "$FM_SUP_NEEDED" = false ]; then + # Leave the unhealthy state (no supervision need): clear so a later need plus + # stale watcher is a fresh episode even if the beacon is still absent with the + # same key string. [ "$READ_ONLY" -eq 1 ] || fm_guard_clear_stale_banner exit 0 fi [ -s "$FM_WAKE_QUEUE" ] && queue_pending=true -# No fresh watcher with tasks in flight is the dangerous state: emit a prominent, -# bordered banner FIRST so it reads as an alarm, not a buried stderr line. Later -# calls in the same episode get a one-line reminder only. +# A supervision need with no fresh watcher is the dangerous state: emit a +# prominent, bordered banner FIRST so it reads as an alarm, not a buried stderr +# line. Later calls in the same episode get a one-line reminder only. if [ "$watcher_fresh" = false ]; then episode_key=$(fm_guard_stale_episode_key "$STATE") episode_key=${episode_key%$'\n'} @@ -187,7 +186,13 @@ if [ "$watcher_fresh" = false ]; then { printf '●%s\n' "$rule" printf '● WATCHER DOWN - SUPERVISION IS OFF\n' - printf '● %s task(s) in flight, but no watcher has a fresh beacon (last beat: %s, grace %ss).\n' "$in_flight" "$beacon_desc" "$GRACE" + if [ "$in_flight" -gt 0 ]; then + printf '● %s task(s) in flight, but no watcher has a fresh beacon (last beat: %s, grace %ss).\n' "$in_flight" "$beacon_desc" "$GRACE" + elif [ "$FM_SUP_REFILL_NEEDED" = true ]; then + printf '● Refill work is pending, but no watcher has a fresh beacon (last beat: %s, grace %ss).\n' "$beacon_desc" "$GRACE" + else + printf '● X-mode relay polling is active, but no watcher has a fresh beacon (last beat: %s, grace %ss).\n' "$beacon_desc" "$GRACE" + fi if [ "$READ_ONLY" -eq 1 ]; then printf '● This read-only session should report the lapse, not repair it.\n' else @@ -202,14 +207,15 @@ if [ "$watcher_fresh" = false ]; then "$beacon_desc" "$GRACE" >&2 fi else - # Healthy again while work is still in flight: end the episode so a later + # Healthy again while supervision remains needed: end the episode so a later # restale re-prints the full banner. [ "$READ_ONLY" -eq 1 ] || fm_guard_clear_stale_banner fi -# Queued wakes are an independent hazard; warn whenever they are pending, even if -# a watcher is alive. Kept after the banner so the no-watcher alarm reads first. -# Dedup of the watcher-down banner never suppresses this warning. +# Queued wakes are independent of watcher freshness; while supervision remains +# needed, warn whenever they are pending even if a watcher is alive. Keep this +# after the banner so the no-watcher alarm reads first. Dedup of the watcher-down +# banner never suppresses this warning. if "$queue_pending"; then if [ "$READ_ONLY" -eq 1 ]; then echo "WARNING: queued wakes pending - left untouched because this session lacks verified fleet-lock ownership." >&2 diff --git a/bin/fm-refill-complete.sh b/bin/fm-refill-complete.sh new file mode 100755 index 0000000000..c87c12379b --- /dev/null +++ b/bin/fm-refill-complete.sh @@ -0,0 +1,27 @@ +#!/usr/bin/env bash +# Record a handled refill claim-and-dispatch cycle that spawned no ship. +set -u + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +FM_ROOT="${FM_ROOT_OVERRIDE:-$(cd "$SCRIPT_DIR/.." && pwd)}" +FM_HOME="${FM_HOME:-${FM_ROOT_OVERRIDE:-$FM_ROOT}}" +STATE="${FM_STATE_OVERRIDE:-$FM_HOME/state}" +CONFIG="${FM_CONFIG_OVERRIDE:-$FM_HOME/config}" + +# shellcheck source=bin/fm-refill-lib.sh +. "$SCRIPT_DIR/fm-refill-lib.sh" + +case "${1:-}" in + no-ready|no-eligible|held-only) outcome=$1 ;; + *) + echo "usage: fm-refill-complete.sh " >&2 + exit 2 + ;; +esac +[ "$#" -eq 1 ] || { + echo "usage: fm-refill-complete.sh " >&2 + exit 2 +} + +fm_refill_dispatch_cycle_completed "$STATE" "$CONFIG" +printf 'refill claim-and-dispatch cycle completed: %s\n' "$outcome" diff --git a/bin/fm-refill-lib.sh b/bin/fm-refill-lib.sh new file mode 100755 index 0000000000..fe3cb1aaf6 --- /dev/null +++ b/bin/fm-refill-lib.sh @@ -0,0 +1,401 @@ +#!/usr/bin/env bash +# shellcheck disable=SC2034 +# Shared completion-triggered refill, concurrency-floor signalling, and refill +# supervision ownership. +# +# Completion and floor events only signal the normal claim-and-dispatch path. +# They never spawn, claim, or bypass pickup checks, exclusions, holds, or parks. +# A home remains supervision-active while a refill wake is queued, a completion +# or floor refill remains unhandled, a completion receipt is pending, or an +# enabled concurrency floor is below target. + +_FM_REFILL_LIB_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +# shellcheck source=bin/fm-wake-lib.sh +. "$_FM_REFILL_LIB_DIR/fm-wake-lib.sh" + +FM_REFILL_CONFIG="${FM_CONFIG_OVERRIDE:-$FM_HOME/config}" +FM_REFILL_REASON= +FM_REFILL_HAZARD= + +fm_refill_backend_load() { + if declare -F fm_backend_of_meta >/dev/null \ + && declare -F fm_backend_target_of_meta >/dev/null \ + && declare -F fm_backend_target_exists >/dev/null \ + && declare -F fm_backend_agent_state >/dev/null; then + return 0 + fi + [ -f "$_FM_REFILL_LIB_DIR/fm-backend.sh" ] \ + && [ ! -L "$_FM_REFILL_LIB_DIR/fm-backend.sh" ] || return 1 + # shellcheck source=bin/fm-backend.sh + . "$_FM_REFILL_LIB_DIR/fm-backend.sh" +} + +fm_refill_task_id_valid() { + local id=$1 + case "$id" in + ''|.*|*[!A-Za-z0-9._-]*) return 1 ;; + esac + [ "${#id}" -le 64 ] +} + +fm_refill_normalize_count() { + local value=$1 + value=${value#"${value%%[!0]*}"} + [ -n "$value" ] || value=0 + printf '%s\n' "$value" +} + +fm_refill_count_below_target() { + local live floor + live=$(fm_refill_normalize_count "$1") + floor=$(fm_refill_normalize_count "$2") + [ "${#live}" -lt "${#floor}" ] && return 0 + [ "${#live}" -gt "${#floor}" ] && return 1 + [[ "$live" < "$floor" ]] +} + +fm_refill_concurrency_floor() { + local config=${1:-$FM_REFILL_CONFIG} f line lines + f="$config/concurrency-floor" + if [ ! -f "$f" ] || [ -L "$f" ]; then + printf '0\n' + return 0 + fi + lines=$(awk 'END { print NR + 0 }' "$f" 2>/dev/null) || lines=0 + [ "$lines" = 1 ] || { printf '0\n'; return 0; } + line=$(head -n 1 "$f" 2>/dev/null) || line= + line=${line#"${line%%[![:space:]]*}"} + line=${line%"${line##*[![:space:]]}"} + case "$line" in + ''|0|*[!0-9]*) printf '0\n' ;; + *) fm_refill_normalize_count "$line" ;; + esac +} + +fm_refill_live_ship_count() { + local state=${1:-$STATE} meta id kind backend target agent_state verdict n=0 + local reader=${FM_CREW_STATE_BIN:-$_FM_REFILL_LIB_DIR/fm-crew-state.sh} + if ! fm_refill_backend_load; then + printf '0\n' + return 0 + fi + shopt -s nullglob + for meta in "$state"/*.meta; do + [ -f "$meta" ] && [ ! -L "$meta" ] || continue + id=$(basename "$meta" .meta) + fm_refill_task_id_valid "$id" || continue + kind=$(grep '^kind=' "$meta" 2>/dev/null | tail -1 | cut -d= -f2- || true) + [ -n "$kind" ] || kind=ship + [ "$kind" = ship ] || continue + backend=$(fm_backend_of_meta "$meta") + target=$(fm_backend_target_of_meta "$meta") + [ -n "$target" ] || continue + fm_backend_target_exists "$backend" "$target" "fm-$id" || continue + agent_state=$(fm_backend_agent_state "$backend" "$target") || agent_state=unreadable + case "$agent_state" in dead|missing) continue ;; esac + verdict=$(FM_HOME="$(dirname "$state")" FM_STATE_OVERRIDE="$state" \ + FM_CREW_STATE_NM_TIMEOUT="${FM_REFILL_CREW_STATE_NM_TIMEOUT:-2}" \ + "$reader" "$id" 2>/dev/null) || verdict= + case "$verdict" in + 'state: working · '*) n=$((n + 1)) ;; + esac + done + shopt -u nullglob + printf '%s\n' "$n" +} + +fm_refill_has_parked_unpushed() { + local state=${1:-$STATE} meta id kind verdict wt unpushed + local reader=${FM_CREW_STATE_BIN:-$_FM_REFILL_LIB_DIR/fm-crew-state.sh} + FM_REFILL_HAZARD= + shopt -s nullglob + for meta in "$state"/*.meta; do + if [ ! -f "$meta" ] || [ -L "$meta" ]; then + FM_REFILL_HAZARD="unreadable metadata state for $(basename "$meta" .meta)" + shopt -u nullglob + return 0 + fi + id=$(basename "$meta" .meta) + if ! fm_refill_task_id_valid "$id"; then + FM_REFILL_HAZARD="unreadable lifecycle state for $id" + shopt -u nullglob + return 0 + fi + kind=$(grep '^kind=' "$meta" 2>/dev/null | tail -1 | cut -d= -f2- || true) + [ -n "$kind" ] || kind=ship + [ "$kind" = ship ] || continue + verdict=$(FM_HOME="$(dirname "$state")" FM_STATE_OVERRIDE="$state" \ + FM_CREW_STATE_NM_TIMEOUT="${FM_REFILL_CREW_STATE_NM_TIMEOUT:-2}" \ + "$reader" "$id" 2>/dev/null) || verdict= + case "$verdict" in + 'state: parked · '*) ;; + 'state: working · '*|'state: done · '*|'state: blocked · '*|\ + 'state: paused · '*|'state: failed · '*) continue ;; + *) + FM_REFILL_HAZARD="unreadable lifecycle state for $id" + shopt -u nullglob + return 0 + ;; + esac + wt=$(grep '^worktree=' "$meta" 2>/dev/null | tail -1 | cut -d= -f2- || true) + if [ -z "$wt" ] || [ ! -d "$wt" ]; then + FM_REFILL_HAZARD="unknown worktree state for $(basename "$meta" .meta)" + shopt -u nullglob + return 0 + fi + if ! unpushed=$(git -C "$wt" log --oneline HEAD --not --remotes -- 2>/dev/null); then + FM_REFILL_HAZARD="unreadable git state for $(basename "$meta" .meta)" + shopt -u nullglob + return 0 + fi + if [ -n "$unpushed" ]; then + FM_REFILL_HAZARD="unpushed commits in $(basename "$meta" .meta)" + shopt -u nullglob + return 0 + fi + done + shopt -u nullglob + return 1 +} + +fm_refill_hold_suffix() { + local state=${1:-$STATE} + if fm_refill_has_parked_unpushed "$state"; then + printf '%s' "; HOLD: parked ship worktree safety is not proven ($FM_REFILL_HAZARD) - do not pool-dispatch until leases protect ship worktrees or the state is proven clear" + fi +} + +fm_refill_handled_suffix() { + printf '%s' '; after the claim-and-dispatch attempt, a successful ship spawn clears any completion need automatically; otherwise run bin/fm-refill-complete.sh ' +} + +fm_refill_completion_marker() { + printf '%s/.refill-completion-%s\n' "${2:-$STATE}" "$1" +} + +fm_refill_completion_needed_marker() { + printf '%s/.refill-needed-completion\n' "${1:-$STATE}" +} + +fm_refill_floor_needed_marker() { + printf '%s/.refill-needed-floor\n' "${1:-$STATE}" +} + +fm_refill_atomic_write() { + local path=$1 value=$2 dir tmp + dir=$(dirname "$path") + mkdir -p "$dir" || return 1 + tmp=$(mktemp "$dir/.refill-write.XXXXXX") || return 1 + if ! (umask 077; printf '%s\n' "$value" > "$tmp") || ! mv -f "$tmp" "$path"; then + rm -f "$tmp" 2>/dev/null || true + return 1 + fi +} + +fm_refill_mark_needed() { + fm_refill_atomic_write "$1" "needed $(date +%s)" +} + +fm_refill_marker_state() { + local marker=$1 value + [ -e "$marker" ] || { printf 'absent\n'; return 0; } + [ -f "$marker" ] && [ ! -L "$marker" ] || { printf 'pending\n'; return 0; } + value=$(head -n 1 "$marker" 2>/dev/null || true) + case "$value" in + pending*) printf 'pending\n' ;; + committed*) printf 'committed\n' ;; + ''|*[!0-9]*) printf 'pending\n' ;; + *) printf 'committed\n' ;; + esac +} + +fm_refill_queue_payload_locked() { + local key=$1 queue=${2:-$FM_WAKE_QUEUE} + awk -F '\t' -v key="$key" '$3 == "check" && $4 == key { payload = $5 } END { if (payload != "") print payload; else exit 1 }' \ + "$queue" 2>/dev/null +} + +fm_refill_queue_any_payload_locked() { + local queue=${1:-$FM_WAKE_QUEUE} + awk -F '\t' '$3 == "check" && $4 ~ /^refill([:-]|$)/ { payload = $5 } END { if (payload != "") print payload; else exit 1 }' \ + "$queue" 2>/dev/null +} + +fm_refill_commit_completion_marker() { + local id=$1 state=${2:-$STATE} + fm_refill_atomic_write "$(fm_refill_completion_marker "$id" "$state")" "committed $(date +%s)" +} + +fm_refill_finalize_completion_receipts() { + local queue=${1:-$FM_WAKE_QUEUE} state=${2:-$STATE} key id + while IFS= read -r key; do + [ -n "$key" ] || continue + id=${key#refill:} + fm_refill_task_id_valid "$id" || return 1 + fm_refill_commit_completion_marker "$id" "$state" || return 1 + fm_refill_mark_needed "$(fm_refill_completion_needed_marker "$state")" || return 1 + done < <(awk -F '\t' '$3 == "check" && $4 ~ /^refill:/ && !seen[$4]++ { print $4 }' "$queue" 2>/dev/null) +} + +fm_refill_recover_pending_completions() { + local state=${1:-$STATE} marker id + shopt -s nullglob + for marker in "$state"/.refill-completion-*; do + [ "$(fm_refill_marker_state "$marker")" = pending ] || continue + id=${marker##*/.refill-completion-} + fm_refill_task_id_valid "$id" || continue + fm_refill_emit_completion "$id" >/dev/null 2>&1 || true + done + shopt -u nullglob +} + +fm_refill_emit_floor_if_needed() { + local state=${1:-$STATE} config=${2:-$FM_REFILL_CONFIG} floor live payload existing status=1 + FM_REFILL_REASON= + floor=$(fm_refill_concurrency_floor "$config") + case "$floor" in + ''|0|*[!0-9]*) + fm_lock_acquire_wait "$FM_WAKE_QUEUE_LOCK" + rm -f "$(fm_refill_floor_needed_marker "$state")" 2>/dev/null || true + fm_lock_release "$FM_WAKE_QUEUE_LOCK" + return 1 + ;; + esac + live=$(fm_refill_live_ship_count "$state") + case "$live" in ''|*[!0-9]*) live=0 ;; esac + if ! fm_refill_count_below_target "$live" "$floor"; then + fm_lock_acquire_wait "$FM_WAKE_QUEUE_LOCK" + rm -f "$(fm_refill_floor_needed_marker "$state")" 2>/dev/null || true + fm_lock_release "$FM_WAKE_QUEUE_LOCK" + return 1 + fi + payload="check: refill floor: live ships $live below target $floor; run normal claim-and-dispatch (tasks-axi ready, date gates, exclusions, held and parked); do not blind-spawn$(fm_refill_handled_suffix)$(fm_refill_hold_suffix "$state")" + fm_lock_acquire_wait "$FM_WAKE_QUEUE_LOCK" + fm_refill_mark_needed "$(fm_refill_floor_needed_marker "$state")" || { + fm_lock_release "$FM_WAKE_QUEUE_LOCK" + return 1 + } + existing=$(fm_refill_queue_payload_locked refill-floor 2>/dev/null || true) + if [ -n "$existing" ]; then + FM_REFILL_REASON=$existing + elif fm_wake_append_locked check refill-floor "$payload"; then + FM_REFILL_REASON=$payload + status=0 + fi + fm_lock_release "$FM_WAKE_QUEUE_LOCK" + return "$status" +} + +fm_refill_emit_completion() { + local id=$1 marker key payload existing marker_state status=1 + FM_REFILL_REASON= + fm_refill_task_id_valid "$id" || return 1 + marker=$(fm_refill_completion_marker "$id") + key="refill:$id" + payload="check: refill completion $id: run normal claim-and-dispatch (tasks-axi ready, date gates, exclusions, held and parked); do not blind-spawn$(fm_refill_handled_suffix)$(fm_refill_hold_suffix "$STATE")" + fm_lock_acquire_wait "$FM_WAKE_QUEUE_LOCK" + marker_state=$(fm_refill_marker_state "$marker") + if [ "$marker_state" = committed ]; then + fm_lock_release "$FM_WAKE_QUEUE_LOCK" + fm_refill_emit_floor_if_needed >/dev/null 2>&1 || true + return 1 + fi + if [ "$marker_state" = absent ]; then + fm_refill_atomic_write "$marker" "pending $(date +%s)" || { + fm_lock_release "$FM_WAKE_QUEUE_LOCK" + return 1 + } + fi + fm_refill_mark_needed "$(fm_refill_completion_needed_marker)" || { + fm_lock_release "$FM_WAKE_QUEUE_LOCK" + return 1 + } + if [ "${FM_REFILL_TEST_STOP_AFTER_PENDING:-0}" = 1 ]; then + fm_lock_release "$FM_WAKE_QUEUE_LOCK" + return 75 + fi + existing=$(fm_refill_queue_payload_locked "$key" 2>/dev/null || true) + if [ -n "$existing" ]; then + fm_refill_commit_completion_marker "$id" || { + fm_lock_release "$FM_WAKE_QUEUE_LOCK" + return 1 + } + FM_REFILL_REASON=$existing + elif fm_wake_append_locked check "$key" "$payload"; then + fm_refill_commit_completion_marker "$id" || { + fm_lock_release "$FM_WAKE_QUEUE_LOCK" + return 1 + } + FM_REFILL_REASON=$payload + status=0 + fi + fm_lock_release "$FM_WAKE_QUEUE_LOCK" + fm_refill_emit_floor_if_needed >/dev/null 2>&1 || true + [ "$status" -ne 0 ] || FM_REFILL_REASON=$payload + return "$status" +} + +fm_refill_pending_marker_exists() { + local state=${1:-$STATE} marker id + [ -f "$(fm_refill_completion_needed_marker "$state")" ] && return 0 + [ -f "$(fm_refill_floor_needed_marker "$state")" ] && return 0 + shopt -s nullglob + for marker in "$state"/.refill-completion-*; do + [ "$(fm_refill_marker_state "$marker")" = pending ] || continue + id=${marker##*/.refill-completion-} + fm_refill_task_id_valid "$id" || continue + shopt -u nullglob + return 0 + done + shopt -u nullglob + return 1 +} + +fm_refill_supervision_needed() { + local state=${1:-$STATE} config=${2:-$FM_REFILL_CONFIG} floor live + fm_refill_queue_any_payload_locked "$state/.wake-queue" >/dev/null 2>&1 && return 0 + fm_refill_pending_marker_exists "$state" && return 0 + floor=$(fm_refill_concurrency_floor "$config") + case "$floor" in ''|0|*[!0-9]*) return 1 ;; esac + live=$(fm_refill_live_ship_count "$state") + case "$live" in ''|*[!0-9]*) live=0 ;; esac + fm_refill_count_below_target "$live" "$floor" +} + +fm_refill_surface_pending_if_needed() { + local payload existing + FM_REFILL_REASON= + fm_refill_recover_pending_completions "$STATE" + fm_lock_acquire_wait "$FM_WAKE_QUEUE_LOCK" + existing=$(fm_refill_queue_any_payload_locked 2>/dev/null || true) + if [ -n "$existing" ]; then + FM_REFILL_REASON=$existing + fm_lock_release "$FM_WAKE_QUEUE_LOCK" + return 0 + fi + if ! fm_refill_pending_marker_exists "$STATE"; then + fm_lock_release "$FM_WAKE_QUEUE_LOCK" + return 1 + fi + payload="check: refill pending: run normal claim-and-dispatch (tasks-axi ready, date gates, exclusions, held and parked); do not blind-spawn$(fm_refill_handled_suffix)$(fm_refill_hold_suffix "$STATE")" + if fm_wake_append_locked check refill-pending "$payload"; then + FM_REFILL_REASON=$payload + fm_lock_release "$FM_WAKE_QUEUE_LOCK" + return 0 + fi + fm_lock_release "$FM_WAKE_QUEUE_LOCK" + return 1 +} + +fm_refill_emit_pending_if_needed() { + fm_refill_emit_floor_if_needed >/dev/null 2>&1 || true + fm_refill_surface_pending_if_needed +} + +fm_refill_dispatch_cycle_completed() { + local state=${1:-$STATE} config=${2:-$FM_REFILL_CONFIG} + fm_lock_acquire_wait "$FM_WAKE_QUEUE_LOCK" + rm -f "$(fm_refill_completion_needed_marker "$state")" 2>/dev/null || true + fm_lock_release "$FM_WAKE_QUEUE_LOCK" + fm_refill_emit_floor_if_needed "$state" "$config" >/dev/null 2>&1 || true +} diff --git a/bin/fm-session-start.sh b/bin/fm-session-start.sh index e4f21d0358..6df872666b 100755 --- a/bin/fm-session-start.sh +++ b/bin/fm-session-start.sh @@ -299,6 +299,9 @@ if [ "$READ_ONLY" -eq 1 ]; then GUARD_OUT=$(FM_GUARD_READ_ONLY=1 "$SCRIPT_DIR/fm-guard.sh" 2>&1) [ -n "$GUARD_OUT" ] && printf '%s\n' "$GUARD_OUT" else + # shellcheck source=bin/fm-refill-lib.sh + . "$SCRIPT_DIR/fm-refill-lib.sh" + fm_refill_emit_pending_if_needed >/dev/null 2>&1 || true DRAIN_OUT=$("$SCRIPT_DIR/fm-wake-drain.sh" 2>&1) if [ -n "$DRAIN_OUT" ]; then printf '%s\n' "$DRAIN_OUT" diff --git a/bin/fm-spawn.sh b/bin/fm-spawn.sh index b6ffdfd347..81c8d767d6 100755 --- a/bin/fm-spawn.sh +++ b/bin/fm-spawn.sh @@ -167,6 +167,8 @@ SUB_HOME_MARKER=".fm-secondmate-home" . "$SCRIPT_DIR/fm-ff-lib.sh" # shellcheck source=bin/fm-wake-lib.sh . "$SCRIPT_DIR/fm-wake-lib.sh" +# shellcheck source=bin/fm-refill-lib.sh +. "$SCRIPT_DIR/fm-refill-lib.sh" # shellcheck source=bin/fm-config-inherit-lib.sh . "$SCRIPT_DIR/fm-config-inherit-lib.sh" # shellcheck source=bin/fm-backend.sh @@ -1722,4 +1724,5 @@ if [ "$KIND" = secondmate ]; then fi fi +[ "$KIND" != ship ] || fm_refill_dispatch_cycle_completed "$STATE" "$CONFIG" echo "spawned $ID harness=$HARNESS kind=$KIND mode=$MODE yolo=$YOLO window=$META_WINDOW worktree=$WT" diff --git a/bin/fm-supervision-lib.sh b/bin/fm-supervision-lib.sh index 1930700d2a..df83c0bec1 100644 --- a/bin/fm-supervision-lib.sh +++ b/bin/fm-supervision-lib.sh @@ -3,14 +3,17 @@ # Usage: . bin/fm-supervision-lib.sh # # Reports whether a firstmate home needs supervision because it has in-flight -# work (a state/.meta exists) or an X-mode relay poll -# (state/x-watch.check.sh), and whether its watcher has a fresh liveness beacon +# work, an X-mode relay poll, or refill work, and whether its watcher has a fresh liveness beacon # (state/.last-watcher-beat, touched every poll cycle, within the grace window). # bin/fm-guard.sh keeps its task-specific grace-based warning predicate; # bin/fm-turnend-guard.sh uses the status fields here for its banner but performs # its end-of-turn block decision with the live watcher lock check in # bin/fm-wake-lib.sh. +_FM_SUP_LIB_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +# shellcheck source=bin/fm-refill-lib.sh +. "$_FM_SUP_LIB_DIR/fm-refill-lib.sh" + # Portable mtime; Linux stat lacks -f, macOS stat lacks -c. fm_sup_stat_mtime() { if [ "$(uname)" = Darwin ]; then @@ -23,10 +26,11 @@ fm_sup_stat_mtime() { # fm_supervision_status [grace-seconds] # Populates, for the state dir at $1: # FM_SUP_IN_FLIGHT count of state/*.meta (in-flight tasks) -# FM_SUP_NEEDED true/false - in-flight work or an X-mode relay poll +# FM_SUP_NEEDED true/false - in-flight work, X-mode relay, or refill work # FM_SUP_WATCHER_FRESH true/false - a watcher beacon within the grace window # FM_SUP_BEACON_DESC human-readable beacon age, for banners ("never" if absent) # FM_SUP_QUEUE_PENDING true/false - state/.wake-queue has unread records +# FM_SUP_REFILL_NEEDED true/false - refill queue, marker, or floor needs a cycle # grace-seconds defaults to $FM_GUARD_GRACE, then 300, matching fm-guard.sh. # Always returns 0; callers read the vars, or use fm_supervision_unhealthy below. fm_supervision_status() { @@ -36,12 +40,16 @@ fm_supervision_status() { FM_SUP_WATCHER_FRESH=false FM_SUP_BEACON_DESC=never FM_SUP_QUEUE_PENDING=false + FM_SUP_REFILL_NEEDED=false for meta in "$state"/*.meta; do [ -e "$meta" ] || continue FM_SUP_IN_FLIGHT=$((FM_SUP_IN_FLIGHT + 1)) done - if [ "$FM_SUP_IN_FLIGHT" -gt 0 ] || [ -f "$state/x-watch.check.sh" ]; then + if fm_refill_supervision_needed "$state" "${FM_CONFIG_OVERRIDE:-$(dirname "$state")/config}"; then + FM_SUP_REFILL_NEEDED=true + fi + if [ "$FM_SUP_IN_FLIGHT" -gt 0 ] || [ -f "$state/x-watch.check.sh" ] || [ "$FM_SUP_REFILL_NEEDED" = true ]; then FM_SUP_NEEDED=true fi @@ -64,7 +72,7 @@ fm_supervision_status() { } # fm_supervision_needed [grace-seconds] -# Exit 0 (true) exactly when in-flight work or an X-mode relay poll needs a +# Exit 0 (true) exactly when in-flight work, an X-mode relay poll, or refill work needs a # watcher. Exit 1 (false) for an idle home. fm_supervision_needed() { fm_supervision_status "$@" diff --git a/bin/fm-teardown.sh b/bin/fm-teardown.sh index ed1da4fa95..b83c68a8a7 100755 --- a/bin/fm-teardown.sh +++ b/bin/fm-teardown.sh @@ -2,8 +2,9 @@ # Tear down a finished task: return the treehouse worktree, release the Orca # worktree, or retire a secondmate home; kill the recorded runtime endpoint, # clear volatile state, refresh/prune the project's clone for PR-based ship -# tasks, then print a backlog-refresh reminder for ship and scout teardowns -# (a secondmate teardown prints none, since secondmates are not backlog items). +# tasks, then print a backlog-refresh reminder and emit a durable completion +# refill wake for ship and scout teardowns (a secondmate teardown prints none +# and emits no refill, since secondmates are not backlog items). # REFUSES if the worktree holds work that has not LANDED, because cleanup # hard-resets/removes the worktree and kills its processes. Work has landed when it is # reachable from any remote-tracking branch (a fork counts as a remote, so @@ -110,6 +111,8 @@ SUB_HOME_MARKER=".fm-secondmate-home" . "$SCRIPT_DIR/fm-public-followup-lib.sh" # shellcheck source=bin/fm-secondmate-registry-lib.sh . "$SCRIPT_DIR/fm-secondmate-registry-lib.sh" +# shellcheck source=bin/fm-refill-lib.sh +. "$SCRIPT_DIR/fm-refill-lib.sh" if [ "$#" -lt 1 ] || ! fm_task_id_path_safe "$1"; then echo "error: invalid teardown request" >&2 exit 2 @@ -1547,3 +1550,11 @@ if [ "$KIND" != scout ] && [ "$KIND" != secondmate ] && [ "$MODE" != local-only fi echo "teardown $ID complete (window $T, worktree $WT)" backlog_refresh_reminder +# Durable refill: the printed reminder above is not enough on its own - a long +# supervising turn can land work and never reach claim-next. Emit once per task +# (idempotent across merge-poll + teardown + re-escalation) so the wake queue +# surfaces refill without relying on agent memory. Floor check is independent. +if [ "$KIND" != secondmate ]; then + fm_refill_emit_completion "$ID" || true + fm_refill_emit_floor_if_needed || true +fi diff --git a/bin/fm-turnend-guard.sh b/bin/fm-turnend-guard.sh index 2e96fb33e4..531f16d72f 100755 --- a/bin/fm-turnend-guard.sh +++ b/bin/fm-turnend-guard.sh @@ -134,16 +134,9 @@ budget_reset() { } fm_supervision_status "$STATE" "$GRACE" -if [ "$CLAUDE_MODE" -eq 1 ]; then - if [ "$FM_SUP_NEEDED" = false ]; then - budget_reset - exit 0 - fi -else - if [ "$FM_SUP_IN_FLIGHT" -eq 0 ]; then - budget_reset - exit 0 - fi +if [ "$FM_SUP_NEEDED" = false ]; then + budget_reset + exit 0 fi if fm_watcher_healthy "$STATE" "$WATCH" "$GRACE" "$FM_HOME"; then budget_reset @@ -164,6 +157,8 @@ block_stop() { printf '● TURN WOULD END BLIND - SUPERVISION IS OFF\n' if [ "$FM_SUP_IN_FLIGHT" -gt 0 ]; then printf '● %s task(s) in flight, but no live watcher holds this home lock (last beat: %s).\n' "$FM_SUP_IN_FLIGHT" "$FM_SUP_BEACON_DESC" + elif [ "$FM_SUP_REFILL_NEEDED" = true ]; then + printf '● Refill work needs supervision, but no live watcher holds this home lock (last beat: %s).\n' "$FM_SUP_BEACON_DESC" else printf '● X-mode relay polling needs supervision, but no live watcher holds this home lock (last beat: %s).\n' "$FM_SUP_BEACON_DESC" fi @@ -229,6 +224,8 @@ if [ "$COUNT" -gt "$BLOCK_BUDGET" ]; then budget_reset if [ "$FM_SUP_IN_FLIGHT" -gt 0 ]; then NEED_DESC="$FM_SUP_IN_FLIGHT task(s) in flight" + elif [ "$FM_SUP_REFILL_NEEDED" = true ]; then + NEED_DESC="refill work pending" else NEED_DESC="X-mode relay polling active" fi diff --git a/bin/fm-wake-drain.sh b/bin/fm-wake-drain.sh index c3bf7335c0..687b341581 100755 --- a/bin/fm-wake-drain.sh +++ b/bin/fm-wake-drain.sh @@ -6,6 +6,8 @@ set -u SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" # shellcheck source=bin/fm-wake-lib.sh . "$SCRIPT_DIR/fm-wake-lib.sh" +# shellcheck source=bin/fm-refill-lib.sh +. "$SCRIPT_DIR/fm-refill-lib.sh" DRAIN_TMP= DRAIN_LOCK_HELD=false @@ -18,10 +20,10 @@ RAW_ROWS= # Reuse fm-guard.sh's existing graced, beacon-based alarm (FM_GUARD_GRACE) - do # not duplicate the beacon math. Because the watcher touches its beacon every # poll cycle, a normal fire leaves a recent beacon well inside grace and stays -# silent; only a genuine stale-beyond-grace lapse with work in flight warns. Call -# after the queue is emptied so guard never re-prints its own queued-wakes notice -# for the records this run just drained, and never let a guard hiccup change the -# drain's exit status. +# silent; only a genuine stale-beyond-grace lapse while fleet, relay, or refill +# work needs supervision warns. Call after the queue is emptied so guard never +# re-prints its own queued-wakes notice for the records this run just drained, +# and never let a guard hiccup change the drain's exit status. assert_watcher_liveness() { "$SCRIPT_DIR/fm-guard.sh" || true } @@ -51,6 +53,8 @@ if [ ! -s "$FM_WAKE_QUEUE" ]; then exit 0 fi +fm_refill_finalize_completion_receipts "$FM_WAKE_QUEUE" "$STATE" || exit 1 + DRAIN_TMP="$STATE/.wake-queue.drain.$(fm_current_pid)" rm -f "$DRAIN_TMP" mv "$FM_WAKE_QUEUE" "$DRAIN_TMP" || exit 1 diff --git a/bin/fm-wake-lib.sh b/bin/fm-wake-lib.sh index 8cec58bec1..4a9e047930 100755 --- a/bin/fm-wake-lib.sh +++ b/bin/fm-wake-lib.sh @@ -379,7 +379,7 @@ fm_wake_clean_field() { LC_ALL=C tr '\t\r\n' ' ' } -fm_wake_append() { +fm_wake_append_locked() { local kind=$1 key=$2 payload=$3 clean_key clean_payload epoch seq seq_file status case "$kind" in signal|stale|check|heartbeat) ;; @@ -392,7 +392,6 @@ fm_wake_append() { seq_file="$STATE/.wake-queue.seq" status=0 - fm_lock_acquire_wait "$FM_WAKE_QUEUE_LOCK" seq=$(cat "$seq_file" 2>/dev/null || echo 0) case "$seq" in ''|*[!0-9]*) seq=0 ;; @@ -402,6 +401,14 @@ fm_wake_append() { if [ "$status" -eq 0 ]; then printf '%s\t%s\t%s\t%s\t%s\n' "$epoch" "$seq" "$kind" "$clean_key" "$clean_payload" >> "$FM_WAKE_QUEUE" || status=$? fi + return "$status" +} + +fm_wake_append() { + local status + fm_lock_acquire_wait "$FM_WAKE_QUEUE_LOCK" + fm_wake_append_locked "$@" + status=$? fm_lock_release "$FM_WAKE_QUEUE_LOCK" return "$status" } diff --git a/bin/fm-watch.sh b/bin/fm-watch.sh index 9f17cbc357..ef886eb320 100755 --- a/bin/fm-watch.sh +++ b/bin/fm-watch.sh @@ -46,6 +46,12 @@ # check: rejected unauthenticated PR poll retirement receipts: # invalid pending retirements were preserved without # running a check or removing poll artifacts +# check: refill completion : ... +# durable completion-triggered claim-next (fm-refill-lib) +# check: refill floor: ... +# concurrency-floor top-up when live ships are below target +# check: refill pending: ... +# durable refill need re-surfaced after its earlier wake drained # heartbeat fleet-scan backstop found an unsurfaced captain-relevant # status, unless afk is active # For normal supervision, resume the session-start primary-harness protocol @@ -77,6 +83,8 @@ mkdir -p "$STATE" . "$SCRIPT_DIR/fm-pending-reply-lib.sh" # shellcheck source=bin/fm-busy-lib.sh . "$SCRIPT_DIR/fm-busy-lib.sh" +# shellcheck source=bin/fm-refill-lib.sh +. "$SCRIPT_DIR/fm-refill-lib.sh" WATCH_LOCK="$STATE/.watch.lock" WATCH_PATH="$SCRIPT_DIR/fm-watch.sh" @@ -706,6 +714,10 @@ if ! fm_pr_poll_retirement_recover_all "$STATE" "$SCRIPT_DIR/fm-pr-poll.sh"; the wake "$reason" fi +if fm_refill_emit_pending_if_needed; then + wake "$FM_REFILL_REASON" +fi + while :; do # Self-eviction: if the singleton lock no longer names this process, a second # watcher has taken over (e.g. a transient duplicate from a racy arm). Stand @@ -717,8 +729,13 @@ while :; do exit 0 fi + if fm_refill_surface_pending_if_needed; then + wake "$FM_REFILL_REASON" + fi + # Liveness beacon for fm-guard.sh: a fresh mtime here means a watcher is - # alive. Supervision scripts warn when this goes stale with tasks in flight. + # alive. Supervision scripts warn when it goes stale while fleet, relay, or + # refill work needs supervision. touch "$STATE/.last-watcher-beat" # Parent-owned secondmate pending-reply reconciliation: resolve correlated @@ -775,6 +792,13 @@ while :; do reason="check: $c: $out" fm_wake_append check "$c" "$reason" || exit 1 if [ "$is_pr_poll" -eq 1 ] && [ "$out" = merged ]; then + # Completion-triggered refill: merge is a completion event. Idempotent + # with teardown's emit for the same task id (fm-refill-lib). + kind=$(fm_meta_get "$STATE/$id.meta" kind) + [ -n "$kind" ] || kind=ship + case "$kind" in + ship|scout) fm_refill_emit_completion "$id" || true ;; + esac if fm_pr_poll_retirement_publish "$STATE" "$id" "$SCRIPT_DIR/fm-pr-poll.sh" "$out"; then fm_pr_poll_retirement_recover_one "$STATE" "$id" "$SCRIPT_DIR/fm-pr-poll.sh" \ || triage_log "merged PR poll retirement remains recoverable for $id" @@ -1029,6 +1053,13 @@ EOF hb=$(( HEARTBEAT * (1 << streak) )) [ "$hb" -gt "$HEARTBEAT_MAX" ] && hb=$HEARTBEAT_MAX if [ "$(age_of "$STATE/.last-heartbeat")" -ge "$hb" ]; then + # Concurrency floor: independent of completion events. When live ships are + # below config/concurrency-floor, enqueue a top-up refill and surface it + # even if the heartbeat would otherwise absorb as no-change. + if fm_refill_emit_floor_if_needed; then + touch "$STATE/.last-heartbeat" + wake "$FM_REFILL_REASON" + fi # Triage: in always-on mode a heartbeat is benign unless the cheap fleet-scan # turns up a captain-relevant status the per-wake path missed. Absorb the # no-change case (advance the schedule and back off exactly as wake() would, diff --git a/docs/architecture.md b/docs/architecture.md index 0a06f274a2..5372fb88b0 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -13,6 +13,14 @@ Actionable wakes include captain-relevant status signals, no-verb signals whose Repeated provably-working stale escalations on the same unchanged pane add an escalation count to the wake reason and, at `FM_WEDGE_DEMAND_INSPECT_COUNT`, a `demand-deep-inspection` marker. A busy pane is otherwise exempt from staleness, but only until its latest `state/.turn-ended` marker reaches `FM_BUSY_TURN_MAX_SECS`, or its `state/.meta` spawn record reaches that age before any turn completes; past that bound it is routed through the same wedge escalation, with the identical reason, escalation count, and `demand-deep-inspection` marker, for inspection only - never an automatic interrupt, signal, or restart. Those actionable wakes are written to a durable local queue (`state/.wake-queue`) before detector state advances, so a missed process exit can be recovered by draining the queue. +Completion also enqueues a durable refill wake through `bin/fm-refill-lib.sh`. +Ship/scout merge-poll `merged` and successful ship/scout teardown each use one crash-recoverable per-task receipt and append `check: refill completion :` exactly once after a successful commit, while persistent secondmate merges remain poll-retirement-only. +Queue drain finalizes any pending completion receipt before consuming its row, so a crash between receipt claim and append retries the missing wake while a committed wake remains deduped across restart. +An optional `config/concurrency-floor` target independently appends `check: refill floor:` when authoritative current-state, endpoint-presence, and recovery-grade agent-state reads show fewer live working ships than the target. +Session start, watcher startup, completion, and heartbeat evaluate refill state, every watcher cycle surfaces a newly queued or pending refill, and the shared supervision predicate keeps queued, pending, or below-target homes active with zero task metadata. +Successful ship dispatch clears a handled completion need automatically, `bin/fm-refill-complete.sh` clears it after a handled no-ready, no-eligible, or held-only claim cycle, and meeting or disabling the floor clears its independent need. +Until ship worktree leases protect pool slots, an unreadable ship lifecycle or any parked ship worktree, including `local-only`, that cannot be proven free of unpushed commits fails closed with a dispatch hold in every refill payload. +Both signals only ask the supervising agent to run the ordinary claim-and-dispatch path; they never spawn, claim, or bypass exclusions. When a canonical validated PR poll returns exactly `merged`, the watcher appends that durable notification before publishing a private receipt bound to the poll's registration, bytes, file identities, metadata, provider, URL, and task ID. The receipt makes retirement safely retryable across restarts: fixed-path recovery revalidates the same evidence, removes the runnable check first, removes its registration and data sidecars, removes the receipt last, and preserves task metadata including `pr=` and `pr_head=`. A concurrent replacement remains armed, every non-merged or invalid observation remains unchanged, and retirement never performs task or persistent-secondmate cleanup. @@ -64,10 +72,10 @@ Claude's `bin/fm-claude-stop-autoarm.sh` hook fires on every Stop and, when the [`watcher-continuity.md`](watcher-continuity.md) owns Claude's residual active-turn coverage and watcher-status command-gating boundary. The existing turn-end guard remains the final backstop for all five harness-engine protocols, with pi-signed sharing Pi's protocol and the `--claude` mode cooperating with the auto-arm claim. Its `--restart` mode signals only the watcher recorded in the current home's `state/.watch.lock`, so restarting one home cannot kill sibling secondmate watchers. -A pull-based guard (`bin/fm-guard.sh`) warns through supervision tool output if the primary checkout is tangled, or if tasks are in flight and that watcher stops running or queued wakes are waiting to be drained. +A pull-based guard (`bin/fm-guard.sh`) warns through supervision tool output if the primary checkout is tangled, if fleet, relay, or refill work needs supervision and that watcher stops running, or, while supervision remains needed, if queued wakes are waiting to be drained. The drain script calls that guard after emptying the queue, which avoids repeating the queued-wakes warning for records it just consumed while still warning on stale watcher liveness. It leads with a prominent bordered tangle banner, while `bin/fm-guard.sh` owns the stale-watcher banner/reminder policy so repeated guarded commands stay noisy without reprinting the full watcher-down banner in the same episode. -On every verified primary harness, tracked hook integration gives the primary session a push-based backstop: when work is in flight and no identity-matched watcher lock with a fresh beacon is live, direct Stop hooks block and passive turn-end hooks force one bounded follow-up. +On every verified primary harness, tracked hook integration gives the primary session a push-based backstop: when fleet, relay, or refill work needs supervision and no identity-matched watcher lock with a fresh beacon is live, direct Stop hooks block and passive turn-end hooks force one bounded follow-up. The guard covers the main primary and genuinely marked secondmate homes, exempts child crewmate/scout worktrees, is loop-safe per harness, and is documented in [turnend-guard.md](turnend-guard.md). A presence-gated sub-supervisor (`bin/fm-supervise-daemon.sh`) extends this for walk-away supervision: the `/afk` skill starts it through the tracked foreground helper `bin/fm-afk-start.sh`, after which the watcher reverts to daemon-managed one-shot mode and the daemon self-handles routine wakes in bash. diff --git a/docs/configuration.md b/docs/configuration.md index dd766287ec..062acd419d 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -23,6 +23,24 @@ Wake, watcher, away-mode, and X-specific state mechanics remain with their named `AGENTS.md` retains the run-once and read-once operator rules, lock-refusal safety, installation consent, and direct-report recovery boundaries because those facts apply at every session start. Ordinary dead-direct-report recovery is owned by `stuck-crewmate-recovery`, while persistent-secondmate recovery is owned by `secondmate-provisioning`. +## Ship concurrency floor (config/concurrency-floor) + +`config/concurrency-floor` is an optional local, gitignored file under the effective Firstmate home that sets a target number of concurrent live ship workers for that home. +The file is one non-negative base-10 integer on a single line; only a positive integer enables the floor. +An absent, unreadable, symlinked, empty, zero, or non-numeric value means the floor is off. +When the floor is on, `bin/fm-refill-lib.sh` counts only `kind=ship` workers whose authoritative `bin/fm-crew-state.sh` current state is `working`, whose recorded backend endpoint is present, and whose recovery-grade agent state is not confirmed dead or missing. +Dead, unknown, terminal, parked, paused, scout, and secondmate records do not count as live capacity. +When that live count drops below the target, the library enqueues one durable `check: refill floor:` wake so the supervising agent runs the normal claim-and-dispatch procedure. +The floor is evaluated on locked session start, completion, watcher startup, and watcher heartbeat, while every running watcher cycle cheaply surfaces a newly queued or pending refill. +The floor is independent of completion-triggered refill (merge-poll and teardown). +Both paths exist so a missed rule cannot starve the queue. +Neither path spawns work: they only surface the signal. +Queued refill wakes, pending completion receipts, durable refill-needed markers, and an enabled below-target floor each keep the home supervision-active even when no task metadata remains. +Successful ship spawn clears a handled completion need automatically, `bin/fm-refill-complete.sh ` clears it after the ordinary claim-and-dispatch attempt completes without a spawn, and a met or disabled floor clears the floor need. +Until ship worktree leases protect pool slots, every refill payload includes a hold when any ship has an unreadable lifecycle state or any parked ship worktree, including `local-only`, cannot be read successfully and proven free of unpushed commits. +Coordinate with that lease work rather than reimplementing pool protection here. +This preference is per home and is not part of secondmate inherited configuration. + ## Pi Calm preference (config/calm) The Pi Calm extension stores the captain's home-local presentation choice in gitignored `config/calm` under the effective Firstmate home, resolved from `FM_HOME`, then `FM_ROOT_OVERRIDE`, then the tracked code root derived from the extension path, or under `FM_CONFIG_OVERRIDE` when that test and specialized-setup override is present. diff --git a/docs/scripts.md b/docs/scripts.md index 767980b75a..57a7dab10c 100644 --- a/docs/scripts.md +++ b/docs/scripts.md @@ -65,7 +65,7 @@ The shared no-mistakes gate refusal for fleet lifecycle entrypoints is summarize | `fm-supervise-daemon.sh` | Presence-gated away-mode sub-supervisor: self-handle routine wakes, guard injection by the detected primary harness, escalate batched digests, alert on failed delivery | | `fm-crew-state.sh` | Print one deterministic current-state line for a crew | | `fm-tangle-lib.sh` | Shared default-branch resolution and primary-checkout tangle classification | -| `fm-supervision-lib.sh` | Shared in-flight-work-without-fresh-watcher-beacon predicate | +| `fm-supervision-lib.sh` | Shared fleet, relay, and refill supervision-need and watcher-freshness predicates | | `fm-ff-lib.sh` | Shared guarded fast-forward helper for origin pulls and local secondmate syncs | | `fm-lock-lib.sh` | Shared "is this git lock provably abandoned?" proof used by teardown and fleet-sync | | `fm-config-inherit-lib.sh` | Shared primary-to-secondmate inherited local-material propagation and config-reread delivery | @@ -74,6 +74,8 @@ The shared no-mistakes gate refusal for fleet lifecycle entrypoints is summarize | `fm-vendor-auth-probe.sh`| Run one hard-bounded, non-destructive authentication probe of a named vendor CLI and report the fact | | `fm-wake-drain.sh` | Atomically drain queued watcher wakes, emit bounded best-effort status-event annotations, then assert watcher liveness | | `fm-wake-lib.sh` | Shared durable wake queue, portable locks, and watcher identity/health helpers | +| `fm-refill-lib.sh` | Crash-recoverable completion refill, current-state concurrency floor, and refill supervision ownership | +| `fm-refill-complete.sh` | Clear a handled completion refill after a no-ready, no-eligible, or held-only pickup cycle | | `fm-classify-lib.sh` | Shared captain-relevant and declared-external-wait wake classification vocabulary | | `fm-send.sh` | Send one verified literal line or supported key through the target's recorded backend | | `fm-busy-lib.sh` | Single owner of the semantic busy-state contract: verdicts, source attribution, and per-harness sources | diff --git a/docs/subagent-guard.md b/docs/subagent-guard.md index 47aaf10e0f..344339c9a3 100644 --- a/docs/subagent-guard.md +++ b/docs/subagent-guard.md @@ -367,8 +367,8 @@ tests/fm-subagent-pretool-check.test.sh This change does not close the deeper harness-agnostic defect. Every firstmate guard's in-flight-work branch keys off `state/.meta`, and only `bin/fm-spawn.sh` writes that record. -`bin/fm-supervision-lib.sh` also recognizes an X-mode relay poll as supervision need, but unaccounted primary work still contributes nothing to that predicate. -Without an independent X-mode need, unaccounted primary work therefore reads as idle rather than suspicious. +`bin/fm-supervision-lib.sh` also recognizes X-mode relay polling and refill work as supervision needs, but unaccounted primary work still contributes nothing to that predicate. +Without an independent X-mode or refill need, unaccounted primary work therefore reads as idle rather than suspicious. The durable fix for that class is to make the guards treat "the primary is doing project-shaped work with zero `state/*.meta` files" as a suspicious state rather than an idle one. That would catch this class on any harness, including work created through `Bash`. diff --git a/docs/turnend-guard.md b/docs/turnend-guard.md index 8ee750de39..7ca919d8bb 100644 --- a/docs/turnend-guard.md +++ b/docs/turnend-guard.md @@ -13,7 +13,7 @@ Do not infer this guard's scope, loop safety, or compatibility tradeoffs for tho `bin/fm-guard.sh` is a pull-based warning that runs only when another supervision command invokes it. The turn-end guard closes the remaining gap at the primary's own turn boundary. -When work is in flight and no identity-matched watcher has a fresh beacon, the harness integration must either block the turn end or force one bounded follow-up that uses the recovery instruction from the emitted session-start protocol. +When fleet, relay, or refill work needs supervision and no identity-matched watcher has a fresh beacon, the harness integration must either block the turn end or force one bounded follow-up that uses the recovery instruction from the emitted session-start protocol. The guard remains a backstop; [`watcher-continuity.md`](watcher-continuity.md) owns normal continuity. ## Shared predicate @@ -25,9 +25,9 @@ An unmarked checkout or invalid marker falls through to the git-dir check. That check keeps crewmate and scout linked worktrees inert because their git dir differs from their git common dir. It also requires `AGENTS.md`, `bin/`, and the effective state directory. -For an in-scope primary, the guard counts in-flight work from `state/*.meta`. -The default cross-harness mode exits silently with no work in flight. -Claude's `--claude` mode also treats `state/x-watch.check.sh` as supervision need, so X-mode relay polling remains guarded without an in-flight task. +For an in-scope primary, the guard counts in-flight work from `state/*.meta` and also reads the shared refill supervision predicate. +The default cross-harness mode exits silently when the home has no supervision need. +Every supported primary guard treats `state/x-watch.check.sh` and queued, pending, or below-target refill work as supervision needs without requiring an in-flight task. Otherwise it calls `fm_watcher_healthy [grace-seconds] [home]` from `bin/fm-wake-lib.sh`, the same identity-matched lock and fresh-beacon check used by `bin/fm-watch-arm.sh`. A stale beacon blocks even when a watcher pid is live. A fresh leftover beacon blocks when the lock is missing, dead, or identity-mismatched. @@ -76,7 +76,7 @@ That warning uses `bin/fm-supervision-instructions.sh --repair-line`, so it alwa ## Compatibility limits - Child crewmate and scout worktrees are outside scope. -- A valid secondmate home is in scope; an idle secondmate endpoint with no X-mode relay poll remains healthy because it has no supervision need. +- A valid secondmate home is in scope; an idle secondmate endpoint with no X-mode relay poll or refill need remains healthy because it has no supervision need. - The direct-blocking and bounded passive-follow-up split is limited to the primary integrations listed above. - OpenCode headless mode and untrusted Grok project hooks remain fail-open at the host boundary. - Kimi Code CLI 0.29.1 exposes only global `[[hooks]]` configuration in `~/.kimi-code/config.toml`, including a `Stop` event with snake_case payload fields `hook_event_name`, `session_id`, `cwd`, and `stop_hook_active`. diff --git a/tests/fm-backend.test.sh b/tests/fm-backend.test.sh index 3052ebc422..483e8466b1 100755 --- a/tests/fm-backend.test.sh +++ b/tests/fm-backend.test.sh @@ -143,7 +143,7 @@ resolve_permissive_tmux_kill_ref() { # when this branch changes tmux dispatch semantics. OLD_BIN_UNCHANGED_SIBLINGS="fm-gate-refuse-lib.sh fm-guard.sh fm-lock-lib.sh fm-tasks-axi-lib.sh fm-pr-lib.sh fm-tangle-lib.sh fm-tmux-lib.sh fm-composer-lib.sh fm-wake-lib.sh fm-classify-lib.sh fm-supervision-lib.sh fm-ff-lib.sh fm-config-inherit-lib.sh fm-project-mode.sh fm-harness.sh fm-crew-state.sh fm-decision-hold.sh fm-backend.sh fm-operational-input.sh fm-public-followup-lib.sh fm-secondmate-registry-lib.sh fm-x-lib.sh" # A pull-request merge may add a new main-only dependency that the branch's older baseline does not have yet. -OLD_BIN_OPTIONAL_SIBLINGS="fm-pending-reply-lib.sh" +OLD_BIN_OPTIONAL_SIBLINGS="fm-pending-reply-lib.sh fm-refill-lib.sh" OLD_BIN_REFACTORED="fm-send.sh fm-peek.sh fm-watch.sh fm-spawn.sh fm-teardown.sh fm-marker-lib.sh" build_old_bin() { # -> echoes root dir (root/bin/