From ea264c6565c5d2ea99f858d066eef8f8b0a3b943 Mon Sep 17 00:00:00 2001 From: AG Werschky Date: Mon, 3 Aug 2026 14:18:26 -0600 Subject: [PATCH 1/9] feat: make work refill event-driven with a concurrency floor Completion (merge-poll and teardown) now enqueues one durable refill wake per task instead of relying on a printed reminder the supervisor can miss mid-turn. An optional config/concurrency-floor target independently surfaces a top-up when live ships drop below the target. Both paths only signal claim-and-dispatch; they never spawn and dedupe repeated completion events so re-fires cannot double-dispatch. --- AGENTS.md | 7 +- bin/fm-refill-lib.sh | 206 ++++++++++++++++++++++++++++++++++++ bin/fm-teardown.sh | 15 ++- bin/fm-watch.sh | 16 +++ docs/architecture.md | 4 + docs/configuration.md | 13 +++ docs/scripts.md | 1 + tests/fm-refill.test.sh | 215 ++++++++++++++++++++++++++++++++++++++ tests/fm-teardown.test.sh | 6 ++ 9 files changed, 480 insertions(+), 3 deletions(-) create mode 100755 bin/fm-refill-lib.sh create mode 100755 tests/fm-refill.test.sh diff --git a/AGENTS.md b/AGENTS.md index c1e5272828..83b65a02a5 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -72,6 +72,7 @@ config/secondmate-harness harness the PRIMARY uses to launch SECONDMATE agents, config/backlog-backend backlog backend override; LOCAL, gitignored; absent or "tasks-axi" = default tasks-axi backend, "manual" = force routine backlog updates to hand-editing; inherited by secondmate homes (section 10) config/backend runtime session-provider backend override for new tasks; LOCAL, gitignored; absent = falls through to runtime auto-detection (the runtime firstmate itself is executing inside), then tmux; tmux is the verified reference backend (docs/tmux-backend.md), while herdr, zellij, orca, and cmux are experimental spawn backends (docs/herdr-backend.md, docs/zellij-backend.md, docs/orca-backend.md, docs/cmux-backend.md) - herdr and cmux can also be selected by runtime auto-detection, zellij and orca never are (always explicit), and codex-app is not accepted; see docs/codex-app-backend.md; inherited by secondmate homes under the primary-authoritative contract in secondmate-provisioning config/calm Pi Calm presentation preference; LOCAL, gitignored, and not inherited; see docs/configuration.md "Pi Calm preference" +config/concurrency-floor optional target of concurrent live ships per home; LOCAL, gitignored, absent or 0 = off; see docs/configuration.md "Ship concurrency floor" config/startup-memory-budget primary-authoritative per-home startup-memory budget; LOCAL, gitignored, materialized as 7,500 estimated tokens by locked primary bootstrap and inherited into secondmate homes; see docs/configuration.md "Startup memory budget" config/herdr-presentation-spaces optional presence flag for Herdr's default-off disposable single-task visual projection; LOCAL, gitignored; inherited by secondmate homes; see docs/herdr-backend.md "Optional presentation spaces" config/cmux-socket-password optional cmux control-socket password; LOCAL, gitignored; read fresh on every cmux CLI call and passed through without ever overriding an operator's own ambient CMUX_SOCKET_PASSWORD when absent (docs/cmux-backend.md "Setup") @@ -110,6 +111,7 @@ state/ volatile runtime signals; gitignored public-followup/ generated private transport for promised public replies: commitment registrations, typed terminal-result inbox, accepted/rejected ledgers (section 14; bin/fm-public-followup.sh) x-poll.error x-poll.claim-error generated X-mode relay and offer-claim diagnostic dedupe markers .wake-queue durable queued wakes: epochseqkindkeypayload + .refill-completion- idempotence marker for one completion-refill wake per task (bin/fm-refill-lib.sh) .afk durable away-mode flag; present = sub-supervisor may inject escalations (set by /afk, cleared on user return) .watch.lock .wake-queue.lock watcher singleton and queue serialization locks .claude-autoarm.lock .claude-autoarm-epoch .turnend-claude-blocks Claude Stop auto-arm single-flight, epoch, and guard-budget records; never touch @@ -334,6 +336,7 @@ Tear down a ship task only after landing is confirmed. A teardown refusal for uncommitted or unlanded work is a stop-and-investigate result, never an obstacle to bypass. Never force teardown without explicit discard authority. After successful teardown, record completion, retain only the configured recent Done history, and re-evaluate queued work whose blockers and time gates have cleared. +Teardown and merge-poll completion also emit a durable refill wake (`bin/fm-refill-lib.sh`); treat that wake as the claim-next trigger rather than relying on mid-turn memory of a printed reminder. A secondmate is persistent and an empty queue is healthy. Retire one only on an explicit captain or main-firstmate decision, after loading `secondmate-provisioning`; its home must contain no work under way, and forced discard still requires explicit captain authority. @@ -365,7 +368,9 @@ Handle actionable wakes as follows: 1. For `signal:`, read the listed event lines first, then reconcile current state only where action depends on it. 2. For `stale:`, inspect the recorded endpoint and load `stuck-crewmate-recovery` for a stopped, looping, confused, or unresponsive worker; a deep-inspection reason also requires current-state and validation-log inspection. -3. For `check:`, act on the named poll result, including merges and X-mode events. +3. For `check:`, act on the named poll result, including merges, X-mode events, and refill signals. + A `check: refill completion :` or `check: refill floor:` wake is claim-next capacity, not a blind spawn: run the normal claim-and-dispatch procedure (verify-at-pickup, atomic claim, exclusions, held and parked items, date gates). + Until ship worktree leases land, hold pool dispatch when `fm_refill_has_parked_unpushed` reports parked unpushed commits (`bin/fm-refill-lib.sh`). 4. For `heartbeat:`, review the whole fleet from the structured fleet view, reconcile suspicious tasks and PR state, update the backlog, and never report an unchanged fleet as progress. When any wake reports a merged PR for a project cloned in this home, refresh that clone through the guarded fleet-sync path. diff --git a/bin/fm-refill-lib.sh b/bin/fm-refill-lib.sh new file mode 100755 index 0000000000..c1fc630ffb --- /dev/null +++ b/bin/fm-refill-lib.sh @@ -0,0 +1,206 @@ +#!/usr/bin/env bash +# Shared completion-triggered refill and concurrency-floor signalling. +# +# OWNERSHIP: this library is the single owner of: +# - durable completion-refill wakes (kind=check, key=refill:) +# - concurrency-floor top-up wakes (kind=check, key=refill-floor) +# - per-task completion idempotence markers (state/.refill-completion-) +# - config/concurrency-floor parsing +# - the pre-lease parked-unpushed hazard probe used before top-up dispatch +# +# PROBLEM: continuous refill was a rule the supervising agent had to remember +# mid-turn. Long single turns that landed work never reached claim-next, so the +# queue starved. Completion events already existed (merge-poll wake, teardown +# backlog reminder) but neither emitted a durable signal the agent must act on. +# +# CONTRACT: +# - Emits durable wake-queue records only. Never spawns, claims, or edits the +# backlog. The supervising agent still runs the normal claim-and-dispatch +# procedure (verify-at-pickup, atomic claim, exclusions, held/parked). +# - Completion emit is idempotent per task id: a second delivery of the same +# completion event (merge re-fire, teardown retry, afk re-escalation) does +# not enqueue a second refill wake. +# - Floor emit is independent of completion: when live ship count is below +# config/concurrency-floor, one top-up wake is enqueued (wake-queue key +# dedupe collapses duplicates still pending). Absent/0/invalid config = off. +# - Until ship worktree leases land (fix-lease-ship-worktrees), dispatch must +# hold while any live ship worktree has unpushed commits; this library only +# probes that hazard and never reimplements pool leasing. +# +# Usage (source from bin/ scripts): +# . "$SCRIPT_DIR/fm-refill-lib.sh" +# fm_refill_emit_completion # 0 if newly emitted, 1 if deduped/skip +# fm_refill_emit_floor_if_needed # 0 if newly emitted, 1 if no top-up +# fm_refill_concurrency_floor # prints non-negative integer target +# fm_refill_live_ship_count # prints current live ship meta count +# fm_refill_has_parked_unpushed # 0 when hazard present, 1 when clear +# +# FM_REFILL_REASON is set to the wake payload when an emit returns 0. + +_FM_REFILL_LIB_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +# shellcheck source=bin/fm-wake-lib.sh +. "$_FM_REFILL_LIB_DIR/fm-wake-lib.sh" + +# Prefer the caller's FM_CONFIG_OVERRIDE; otherwise the home's config dir. +# wake-lib already resolved FM_HOME/STATE for this process. +FM_REFILL_CONFIG="${FM_CONFIG_OVERRIDE:-$FM_HOME/config}" + +# Set to the wake payload when an emit returns 0; callers (fm-watch.sh) read it. +# shellcheck disable=SC2034 # External consumer reads this after a successful emit. +FM_REFILL_REASON= + +fm_refill_task_id_valid() { + local id=$1 + case "$id" in + ''|.*|*[!A-Za-z0-9._-]*) return 1 ;; + esac + [ "${#id}" -le 64 ] +} + +# Print the configured concurrency floor (live ships target). Absent file, +# unreadable path, symlink, empty, zero, or non-digits → 0 (feature off). +fm_refill_concurrency_floor() { + local f line + f="$FM_REFILL_CONFIG/concurrency-floor" + if [ ! -f "$f" ] || [ -L "$f" ]; then + printf '0\n' + return 0 + fi + line=$(head -n 1 "$f" 2>/dev/null | tr -d '[:space:]') || line= + case "$line" in + ''|0|*[!0-9]*) + printf '0\n' + ;; + *) + printf '%s\n' "$line" + ;; + esac +} + +# Count live ship workers from state/.meta. Missing kind defaults to ship +# (spawn's historical default). Scouts and secondmates do not count. +fm_refill_live_ship_count() { + local meta id kind n=0 + shopt -s nullglob + for meta in "$STATE"/*.meta; do + [ -f "$meta" ] && [ ! -L "$meta" ] || continue + id=$(basename "$meta" .meta) + fm_refill_task_id_valid "$id" || continue + kind=$(grep '^kind=' "$meta" 2>/dev/null | tail -1 | cut -d= -f2- || true) + [ -n "$kind" ] || kind=ship + case "$kind" in + ship) n=$((n + 1)) ;; + esac + done + shopt -u nullglob + printf '%s\n' "$n" +} + +# 0 when any live ship worktree still has commits not on any remote (the +# pre-lease pool-recycle hazard). local-only ships are excluded: they land via +# local main rather than a remote push. Returns 1 when clear or unreadable. +# Coordinate with fix-lease-ship-worktrees: once ship worktrees are leased, the +# supervising agent can stop treating this probe as a hard hold. +fm_refill_has_parked_unpushed() { + local meta id kind mode wt unpushed + shopt -s nullglob + for meta in "$STATE"/*.meta; do + [ -f "$meta" ] && [ ! -L "$meta" ] || continue + id=$(basename "$meta" .meta) + fm_refill_task_id_valid "$id" || continue + kind=$(grep '^kind=' "$meta" 2>/dev/null | tail -1 | cut -d= -f2- || true) + [ -n "$kind" ] || kind=ship + [ "$kind" = ship ] || continue + mode=$(grep '^mode=' "$meta" 2>/dev/null | tail -1 | cut -d= -f2- || true) + [ -n "$mode" ] || mode=no-mistakes + [ "$mode" != local-only ] || continue + wt=$(grep '^worktree=' "$meta" 2>/dev/null | tail -1 | cut -d= -f2- || true) + [ -n "$wt" ] && [ -d "$wt" ] || continue + unpushed=$(git -C "$wt" log --oneline HEAD --not --remotes -- 2>/dev/null) || continue + if [ -n "$unpushed" ]; then + shopt -u nullglob + return 0 + fi + done + shopt -u nullglob + return 1 +} + +fm_refill_completion_marker() { + local id=$1 + printf '%s/.refill-completion-%s\n' "$STATE" "$id" +} + +# Claim exclusive right to emit one completion refill for . +# Uses noclobber create so concurrent and re-fired deliveries race safely. +# Returns 0 on fresh claim, 1 if already claimed or id invalid. +fm_refill_claim_completion() { + local id=$1 marker + fm_refill_task_id_valid "$id" || return 1 + marker=$(fm_refill_completion_marker "$id") + mkdir -p "$STATE" || return 1 + if ( + set -C + umask 077 + : >"$marker" + ) 2>/dev/null; then + printf '%s\n' "$(date +%s)" >"$marker" || true + return 0 + fi + return 1 +} + +fm_refill_release_completion_claim() { + local id=$1 marker + fm_refill_task_id_valid "$id" || return 0 + marker=$(fm_refill_completion_marker "$id") + rm -f -- "$marker" 2>/dev/null || true +} + +# Emit one durable completion-refill wake for . +# Idempotent: a second call for the same id is a no-op (return 1). +# Does not spawn. On success sets FM_REFILL_REASON and returns 0. +fm_refill_emit_completion() { + local id=$1 key payload + # shellcheck disable=SC2034 # Cleared for callers that read FM_REFILL_REASON after emit. + FM_REFILL_REASON= + fm_refill_task_id_valid "$id" || return 1 + fm_refill_claim_completion "$id" || return 1 + key="refill:$id" + payload="check: refill completion $id: run normal claim-and-dispatch (tasks-axi ready, date gates, exclusions, held and parked); do not blind-spawn" + if ! fm_wake_append check "$key" "$payload"; then + fm_refill_release_completion_claim "$id" + return 1 + fi + # shellcheck disable=SC2034 # Public result consumed by fm-watch.sh after a successful emit. + FM_REFILL_REASON=$payload + return 0 +} + +# Emit one durable floor top-up wake when live ships are below the configured +# target. Absent/zero floor → no-op. At or above target → no-op. Pending queue +# key refill-floor collapses duplicates until drained; a later drop below the +# floor after drain can emit again (independent safety net). +# Returns 0 when newly enqueued, 1 when no top-up is needed or enqueue fails. +fm_refill_emit_floor_if_needed() { + local floor live payload + # shellcheck disable=SC2034 # Cleared for callers that read FM_REFILL_REASON after emit. + FM_REFILL_REASON= + floor=$(fm_refill_concurrency_floor) + case "$floor" in + ''|0|*[!0-9]*) return 1 ;; + esac + live=$(fm_refill_live_ship_count) + case "$live" in + ''|*[!0-9]*) live=0 ;; + esac + [ "$live" -lt "$floor" ] || return 1 + payload="check: refill floor: live ships $live below target $floor; run normal claim-and-dispatch (tasks-axi ready, date gates, exclusions, held and parked); do not blind-spawn" + if fm_refill_has_parked_unpushed; then + payload="$payload; HOLD: parked ship worktree has unpushed commits - do not pool-dispatch until leases protect ship worktrees or unpushed work is cleared" + fi + fm_wake_append check refill-floor "$payload" || return 1 + # shellcheck disable=SC2034 # Public result consumed by fm-watch.sh after a successful emit. + FM_REFILL_REASON=$payload + return 0 +} diff --git a/bin/fm-teardown.sh b/bin/fm-teardown.sh index ed1da4fa95..b83c68a8a7 100755 --- a/bin/fm-teardown.sh +++ b/bin/fm-teardown.sh @@ -2,8 +2,9 @@ # Tear down a finished task: return the treehouse worktree, release the Orca # worktree, or retire a secondmate home; kill the recorded runtime endpoint, # clear volatile state, refresh/prune the project's clone for PR-based ship -# tasks, then print a backlog-refresh reminder for ship and scout teardowns -# (a secondmate teardown prints none, since secondmates are not backlog items). +# tasks, then print a backlog-refresh reminder and emit a durable completion +# refill wake for ship and scout teardowns (a secondmate teardown prints none +# and emits no refill, since secondmates are not backlog items). # REFUSES if the worktree holds work that has not LANDED, because cleanup # hard-resets/removes the worktree and kills its processes. Work has landed when it is # reachable from any remote-tracking branch (a fork counts as a remote, so @@ -110,6 +111,8 @@ SUB_HOME_MARKER=".fm-secondmate-home" . "$SCRIPT_DIR/fm-public-followup-lib.sh" # shellcheck source=bin/fm-secondmate-registry-lib.sh . "$SCRIPT_DIR/fm-secondmate-registry-lib.sh" +# shellcheck source=bin/fm-refill-lib.sh +. "$SCRIPT_DIR/fm-refill-lib.sh" if [ "$#" -lt 1 ] || ! fm_task_id_path_safe "$1"; then echo "error: invalid teardown request" >&2 exit 2 @@ -1547,3 +1550,11 @@ if [ "$KIND" != scout ] && [ "$KIND" != secondmate ] && [ "$MODE" != local-only fi echo "teardown $ID complete (window $T, worktree $WT)" backlog_refresh_reminder +# Durable refill: the printed reminder above is not enough on its own - a long +# supervising turn can land work and never reach claim-next. Emit once per task +# (idempotent across merge-poll + teardown + re-escalation) so the wake queue +# surfaces refill without relying on agent memory. Floor check is independent. +if [ "$KIND" != secondmate ]; then + fm_refill_emit_completion "$ID" || true + fm_refill_emit_floor_if_needed || true +fi diff --git a/bin/fm-watch.sh b/bin/fm-watch.sh index 9f17cbc357..3e2d270f86 100755 --- a/bin/fm-watch.sh +++ b/bin/fm-watch.sh @@ -46,6 +46,10 @@ # check: rejected unauthenticated PR poll retirement receipts: # invalid pending retirements were preserved without # running a check or removing poll artifacts +# check: refill completion : ... +# durable completion-triggered claim-next (fm-refill-lib) +# check: refill floor: ... +# concurrency-floor top-up when live ships are below target # heartbeat fleet-scan backstop found an unsurfaced captain-relevant # status, unless afk is active # For normal supervision, resume the session-start primary-harness protocol @@ -77,6 +81,8 @@ mkdir -p "$STATE" . "$SCRIPT_DIR/fm-pending-reply-lib.sh" # shellcheck source=bin/fm-busy-lib.sh . "$SCRIPT_DIR/fm-busy-lib.sh" +# shellcheck source=bin/fm-refill-lib.sh +. "$SCRIPT_DIR/fm-refill-lib.sh" WATCH_LOCK="$STATE/.watch.lock" WATCH_PATH="$SCRIPT_DIR/fm-watch.sh" @@ -775,6 +781,9 @@ while :; do reason="check: $c: $out" fm_wake_append check "$c" "$reason" || exit 1 if [ "$is_pr_poll" -eq 1 ] && [ "$out" = merged ]; then + # Completion-triggered refill: merge is a completion event. Idempotent + # with teardown's emit for the same task id (fm-refill-lib). + fm_refill_emit_completion "$id" || true if fm_pr_poll_retirement_publish "$STATE" "$id" "$SCRIPT_DIR/fm-pr-poll.sh" "$out"; then fm_pr_poll_retirement_recover_one "$STATE" "$id" "$SCRIPT_DIR/fm-pr-poll.sh" \ || triage_log "merged PR poll retirement remains recoverable for $id" @@ -1029,6 +1038,13 @@ EOF hb=$(( HEARTBEAT * (1 << streak) )) [ "$hb" -gt "$HEARTBEAT_MAX" ] && hb=$HEARTBEAT_MAX if [ "$(age_of "$STATE/.last-heartbeat")" -ge "$hb" ]; then + # Concurrency floor: independent of completion events. When live ships are + # below config/concurrency-floor, enqueue a top-up refill and surface it + # even if the heartbeat would otherwise absorb as no-change. + if fm_refill_emit_floor_if_needed; then + touch "$STATE/.last-heartbeat" + wake "$FM_REFILL_REASON" + fi # Triage: in always-on mode a heartbeat is benign unless the cheap fleet-scan # turns up a captain-relevant status the per-wake path missed. Absorb the # no-change case (advance the schedule and back off exactly as wake() would, diff --git a/docs/architecture.md b/docs/architecture.md index 0a06f274a2..2ba24e8ac4 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -13,6 +13,10 @@ Actionable wakes include captain-relevant status signals, no-verb signals whose Repeated provably-working stale escalations on the same unchanged pane add an escalation count to the wake reason and, at `FM_WEDGE_DEMAND_INSPECT_COUNT`, a `demand-deep-inspection` marker. A busy pane is otherwise exempt from staleness, but only until its latest `state/.turn-ended` marker reaches `FM_BUSY_TURN_MAX_SECS`, or its `state/.meta` spawn record reaches that age before any turn completes; past that bound it is routed through the same wedge escalation, with the identical reason, escalation count, and `demand-deep-inspection` marker, for inspection only - never an automatic interrupt, signal, or restart. Those actionable wakes are written to a durable local queue (`state/.wake-queue`) before detector state advances, so a missed process exit can be recovered by draining the queue. +Completion also enqueues a durable refill wake through `bin/fm-refill-lib.sh`. +Merge-poll `merged` and successful ship/scout teardown each claim one per-task completion marker and append `check: refill completion :`. +An optional `config/concurrency-floor` target independently appends `check: refill floor:` when live ships fall below that count on the heartbeat cadence. +Both signals only ask the supervising agent to run the ordinary claim-and-dispatch path; they never spawn, claim, or bypass exclusions. When a canonical validated PR poll returns exactly `merged`, the watcher appends that durable notification before publishing a private receipt bound to the poll's registration, bytes, file identities, metadata, provider, URL, and task ID. The receipt makes retirement safely retryable across restarts: fixed-path recovery revalidates the same evidence, removes the runnable check first, removes its registration and data sidecars, removes the receipt last, and preserves task metadata including `pr=` and `pr_head=`. A concurrent replacement remains armed, every non-merged or invalid observation remains unchanged, and retirement never performs task or persistent-secondmate cleanup. diff --git a/docs/configuration.md b/docs/configuration.md index dd766287ec..469aff9893 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -23,6 +23,19 @@ Wake, watcher, away-mode, and X-specific state mechanics remain with their named `AGENTS.md` retains the run-once and read-once operator rules, lock-refusal safety, installation consent, and direct-report recovery boundaries because those facts apply at every session start. Ordinary dead-direct-report recovery is owned by `stuck-crewmate-recovery`, while persistent-secondmate recovery is owned by `secondmate-provisioning`. +## Ship concurrency floor (config/concurrency-floor) + +`config/concurrency-floor` is an optional local, gitignored file under the effective Firstmate home that sets a target number of concurrent live ship workers for that home. +The file is one non-negative base-10 integer on a single line; only a positive integer enables the floor. +An absent, unreadable, symlinked, empty, zero, or non-numeric value means the floor is off. +When the floor is on and the count of live `kind=ship` metadata records in `state/` drops below the target, `bin/fm-refill-lib.sh` enqueues one durable `check: refill floor:` wake so the supervising agent runs the normal claim-and-dispatch procedure. +The floor is independent of completion-triggered refill (merge-poll and teardown). +Both paths exist so a missed rule cannot starve the queue. +Neither path spawns work: they only surface the signal. +Until ship worktree leases protect pool slots, a floor or completion refill payload may include a hold when any live non-`local-only` ship worktree still has unpushed commits. +Coordinate with that lease work rather than reimplementing pool protection here. +This preference is per home and is not part of secondmate inherited configuration. + ## Pi Calm preference (config/calm) The Pi Calm extension stores the captain's home-local presentation choice in gitignored `config/calm` under the effective Firstmate home, resolved from `FM_HOME`, then `FM_ROOT_OVERRIDE`, then the tracked code root derived from the extension path, or under `FM_CONFIG_OVERRIDE` when that test and specialized-setup override is present. diff --git a/docs/scripts.md b/docs/scripts.md index 767980b75a..1319235794 100644 --- a/docs/scripts.md +++ b/docs/scripts.md @@ -74,6 +74,7 @@ The shared no-mistakes gate refusal for fleet lifecycle entrypoints is summarize | `fm-vendor-auth-probe.sh`| Run one hard-bounded, non-destructive authentication probe of a named vendor CLI and report the fact | | `fm-wake-drain.sh` | Atomically drain queued watcher wakes, emit bounded best-effort status-event annotations, then assert watcher liveness | | `fm-wake-lib.sh` | Shared durable wake queue, portable locks, and watcher identity/health helpers | +| `fm-refill-lib.sh` | Completion-triggered refill and concurrency-floor top-up wakes (idempotent, no spawn) | | `fm-classify-lib.sh` | Shared captain-relevant and declared-external-wait wake classification vocabulary | | `fm-send.sh` | Send one verified literal line or supported key through the target's recorded backend | | `fm-busy-lib.sh` | Single owner of the semantic busy-state contract: verdicts, source attribution, and per-harness sources | diff --git a/tests/fm-refill.test.sh b/tests/fm-refill.test.sh new file mode 100755 index 0000000000..ef61e47376 --- /dev/null +++ b/tests/fm-refill.test.sh @@ -0,0 +1,215 @@ +#!/usr/bin/env bash +# tests/fm-refill.test.sh - completion-triggered refill and concurrency-floor +# guarantees: idempotent per-task completion emit, durable wake survival, floor +# below/at/above target, absent config = off, and no blind spawn. +set -u + +# shellcheck source=tests/lib.sh +. "$(dirname "${BASH_SOURCE[0]}")/lib.sh" +# shellcheck source=tests/wake-helpers.sh +. "$(dirname "${BASH_SOURCE[0]}")/wake-helpers.sh" + +DRAIN="$ROOT/bin/fm-wake-drain.sh" +TMP_ROOT=$(fm_test_tmproot fm-refill-tests) + +make_refill_home() { + local name=$1 dir + dir=$(make_case "$name") + mkdir -p "$dir/config" "$dir/state" "$dir/data" + printf '%s\n' "$dir" +} + +source_refill() { + local home=$1 + # shellcheck disable=SC1090,SC1091 + FM_HOME="$home" FM_STATE_OVERRIDE="$home/state" FM_CONFIG_OVERRIDE="$home/config" \ + bash -c ' + set -e + . "$1" + shift + "$@" + ' _ "$ROOT/bin/fm-refill-lib.sh" "${@:2}" +} + +queue_refill_rows() { + local state=$1 + awk -F '\t' '$3 == "check" && $4 ~ /^refill/ { print }' "$state/.wake-queue" 2>/dev/null || true +} + +count_refill_keys() { + local state=$1 key=$2 + awk -F '\t' -v k="$key" '$3 == "check" && $4 == k { n++ } END { print n + 0 }' \ + "$state/.wake-queue" 2>/dev/null || printf '0\n' +} + +test_completion_double_fire_emits_once() { + local home state n marker + home=$(make_refill_home completion-once) + state="$home/state" + + source_refill "$home" fm_refill_emit_completion task-a \ + || fail "first completion emit should succeed" + source_refill "$home" fm_refill_emit_completion task-a \ + && fail "second completion emit for same id must be a no-op" || true + + n=$(count_refill_keys "$state" "refill:task-a") + [ "$n" -eq 1 ] || fail "expected exactly one refill:task-a wake, got $n" + marker="$state/.refill-completion-task-a" + [ -f "$marker" ] || fail "completion marker missing after emit" + grep -F 'check: refill completion task-a:' "$state/.wake-queue" >/dev/null \ + || fail "wake payload missing completion claim-next instruction" + grep -F 'do not blind-spawn' "$state/.wake-queue" >/dev/null \ + || fail "wake payload must forbid blind-spawn" + pass "double completion delivery produces exactly one refill signal" +} + +test_completion_signal_is_durable_across_restart() { + local home state drain_out n + home=$(make_refill_home completion-durable) + state="$home/state" + + source_refill "$home" fm_refill_emit_completion task-b \ + || fail "completion emit should succeed" + # Simulate a process restart: re-source the lib and assert the queue still + # holds the wake without a second emission for the same task. + source_refill "$home" fm_refill_emit_completion task-b \ + && fail "restart re-fire must remain deduped" || true + n=$(count_refill_keys "$state" "refill:task-b") + [ "$n" -eq 1 ] || fail "queue must still hold exactly one durable refill after restart re-fire, got $n" + + FM_STATE_OVERRIDE="$state" "$DRAIN" > "$home/drain.out" \ + || fail "drain of durable refill failed" + drain_out=$(cat "$home/drain.out") + printf '%s\n' "$drain_out" | grep -F $'\tcheck\trefill:task-b\t' >/dev/null \ + || fail "drained rows missing durable refill:task-b wake: $drain_out" + [ ! -s "$state/.wake-queue" ] || fail "queue should be empty after drain" + pass "completion refill signal is durable and survives restart re-fire" +} + +test_distinct_tasks_each_get_one_signal() { + local home state n + home=$(make_refill_home completion-distinct) + state="$home/state" + + source_refill "$home" fm_refill_emit_completion task-one || fail "emit one" + source_refill "$home" fm_refill_emit_completion task-two || fail "emit two" + n=$(awk -F '\t' '$3 == "check" && $4 ~ /^refill:/ { n++ } END { print n + 0 }' \ + "$state/.wake-queue") + [ "$n" -eq 2 ] || fail "expected two distinct completion refills, got $n" + pass "distinct completed tasks each produce one refill signal" +} + +test_floor_below_target_emits_top_up() { + local home state n live + home=$(make_refill_home floor-below) + state="$home/state" + printf '2\n' > "$home/config/concurrency-floor" + printf 'window=w1\nkind=ship\n' > "$state/ship-a.meta" + + source_refill "$home" fm_refill_emit_floor_if_needed \ + || fail "floor below target should emit" + n=$(count_refill_keys "$state" refill-floor) + [ "$n" -eq 1 ] || fail "expected one refill-floor wake, got $n" + grep -F 'live ships 1 below target 2' "$state/.wake-queue" >/dev/null \ + || fail "floor payload missing live/target counts" + live=$(source_refill "$home" fm_refill_live_ship_count) + [ "$live" = 1 ] || fail "live ship count should be 1, got $live" + pass "floor emits top-up when live ships are below target" +} + +test_floor_at_or_above_target_silent() { + local home state n + home=$(make_refill_home floor-met) + state="$home/state" + printf '2\n' > "$home/config/concurrency-floor" + printf 'window=w1\nkind=ship\n' > "$state/ship-a.meta" + printf 'window=w2\nkind=ship\n' > "$state/ship-b.meta" + + source_refill "$home" fm_refill_emit_floor_if_needed \ + && fail "floor at target must not emit" || true + printf 'window=w3\nkind=ship\n' > "$state/ship-c.meta" + source_refill "$home" fm_refill_emit_floor_if_needed \ + && fail "floor above target must not emit" || true + n=$(count_refill_keys "$state" refill-floor) + [ "$n" -eq 0 ] || fail "expected no floor wake when at/above target, got $n" + pass "floor is silent when live ships are at or above target" +} + +test_absent_config_floor_off() { + local home state n floor + home=$(make_refill_home floor-absent) + state="$home/state" + # No config/concurrency-floor file. Zero ships would be below any positive + # target, but absent config must keep the feature off. + floor=$(source_refill "$home" fm_refill_concurrency_floor) + [ "$floor" = 0 ] || fail "absent config must parse as 0, got $floor" + source_refill "$home" fm_refill_emit_floor_if_needed \ + && fail "absent config must not emit floor top-up" || true + printf '0\n' > "$home/config/concurrency-floor" + source_refill "$home" fm_refill_emit_floor_if_needed \ + && fail "zero floor must not emit" || true + printf 'nope\n' > "$home/config/concurrency-floor" + source_refill "$home" fm_refill_emit_floor_if_needed \ + && fail "invalid floor must not emit" || true + n=$(count_refill_keys "$state" refill-floor) + [ "$n" -eq 0 ] || fail "expected no floor wakes for safe-absent config, got $n" + pass "absent/zero/invalid concurrency-floor is feature-off" +} + +test_scouts_and_secondmates_do_not_count_as_live_ships() { + local home live + home=$(make_refill_home floor-kinds) + printf 'window=w1\nkind=scout\n' > "$home/state/scout-a.meta" + printf 'window=w2\nkind=secondmate\n' > "$home/state/sm-a.meta" + printf 'window=w3\n' > "$home/state/default-ship.meta" # missing kind = ship + live=$(source_refill "$home" fm_refill_live_ship_count) + [ "$live" = 1 ] || fail "only default-kind ship should count, got $live" + pass "live ship count excludes scout and secondmate meta" +} + +test_parked_unpushed_probe() { + local home wt + home=$(make_refill_home parked-probe) + wt="$home/wt-ship" + mkdir -p "$wt" + git -C "$wt" init -q + git -C "$wt" config user.email "t@example.com" + git -C "$wt" config user.name "t" + printf 'x\n' > "$wt/f" + git -C "$wt" add f + git -C "$wt" commit -qm init + # No remotes → HEAD --not --remotes lists the commit as unpushed. + printf 'window=w1\nkind=ship\nmode=no-mistakes\nworktree=%s\n' "$wt" \ + > "$home/state/parked.meta" + source_refill "$home" fm_refill_has_parked_unpushed \ + || fail "parked unpushed ship should report hazard" + printf '3\n' > "$home/config/concurrency-floor" + source_refill "$home" fm_refill_emit_floor_if_needed \ + || fail "floor should still emit with HOLD note" + grep -F 'HOLD: parked ship worktree has unpushed commits' \ + "$home/state/.wake-queue" >/dev/null \ + || fail "floor payload must include parked-unpushed hold" + pass "parked unpushed probe holds floor payload without blocking the signal" +} + +test_invalid_task_id_refused() { + local home + home=$(make_refill_home bad-id) + source_refill "$home" fm_refill_emit_completion '../escape' \ + && fail "path-like id must be refused" || true + source_refill "$home" fm_refill_emit_completion '' \ + && fail "empty id must be refused" || true + [ ! -e "$home/state/.wake-queue" ] || [ ! -s "$home/state/.wake-queue" ] \ + || fail "invalid ids must not write wakes" + pass "invalid task ids are refused without enqueue" +} + +test_completion_double_fire_emits_once +test_completion_signal_is_durable_across_restart +test_distinct_tasks_each_get_one_signal +test_floor_below_target_emits_top_up +test_floor_at_or_above_target_silent +test_absent_config_floor_off +test_scouts_and_secondmates_do_not_count_as_live_ships +test_parked_unpushed_probe +test_invalid_task_id_refused diff --git a/tests/fm-teardown.test.sh b/tests/fm-teardown.test.sh index 23ef6b8af2..cc0e2a9ff7 100755 --- a/tests/fm-teardown.test.sh +++ b/tests/fm-teardown.test.sh @@ -531,6 +531,12 @@ test_teardown_prompts_tasks_axi_done_when_compatible() { || fail "teardown did not preserve date-gate check: $out" printf '%s\n' "$out" | grep -F 'keep Done to the 10 most recent' >/dev/null \ && fail "teardown kept manual Done pruning in compatible tasks-axi prompt: $out" + # Printed reminder stays, and a durable completion-refill wake is enqueued once. + [ -f "$case_dir/state/.refill-completion-task-x1" ] \ + || fail "teardown did not claim completion-refill marker for task-x1" + awk -F '\t' '$3 == "check" && $4 == "refill:task-x1" { found=1 } END { exit !found }' \ + "$case_dir/state/.wake-queue" \ + || fail "teardown did not enqueue durable refill:task-x1 wake" pass "teardown prompts tasks-axi backlog refresh when compatible" } From bfd6d3c8843e0b0643640683f49ab2f4d50dbb79 Mon Sep 17 00:00:00 2001 From: AG Werschky Date: Mon, 3 Aug 2026 14:45:41 -0600 Subject: [PATCH 2/9] no-mistakes(review): Captain, harden refill supervision and crash recovery --- AGENTS.md | 11 +- bin/fm-claude-stop-autoarm.sh | 6 +- bin/fm-guard.sh | 12 +- bin/fm-refill-lib.sh | 405 ++++++++++++++++++--------- bin/fm-session-start.sh | 3 + bin/fm-spawn.sh | 3 + bin/fm-supervision-lib.sh | 18 +- bin/fm-turnend-guard.sh | 17 +- bin/fm-wake-drain.sh | 4 + bin/fm-wake-lib.sh | 11 +- bin/fm-watch.sh | 6 + docs/architecture.md | 8 +- docs/configuration.md | 9 +- docs/scripts.md | 2 +- docs/turnend-guard.md | 8 +- tests/fm-backend.test.sh | 2 +- tests/fm-claude-stop-autoarm.test.sh | 1 + tests/fm-refill.test.sh | 123 +++++++- tests/fm-turnend-guard.test.sh | 1 + 19 files changed, 482 insertions(+), 168 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 83b65a02a5..7f8dfd84db 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -111,7 +111,8 @@ state/ volatile runtime signals; gitignored public-followup/ generated private transport for promised public replies: commitment registrations, typed terminal-result inbox, accepted/rejected ledgers (section 14; bin/fm-public-followup.sh) x-poll.error x-poll.claim-error generated X-mode relay and offer-claim diagnostic dedupe markers .wake-queue durable queued wakes: epochseqkindkeypayload - .refill-completion- idempotence marker for one completion-refill wake per task (bin/fm-refill-lib.sh) + .refill-completion- crash-recoverable idempotence receipt for one completion-refill wake per task (bin/fm-refill-lib.sh) + .refill-needed-completion .refill-needed-floor durable home-level refill supervision markers (bin/fm-refill-lib.sh) .afk durable away-mode flag; present = sub-supervisor may inject escalations (set by /afk, cleared on user return) .watch.lock .wake-queue.lock watcher singleton and queue serialization locks .claude-autoarm.lock .claude-autoarm-epoch .turnend-claude-blocks Claude Stop auto-arm single-flight, epoch, and guard-budget records; never touch @@ -353,8 +354,8 @@ The promoted worker must inventory scratch state, return to a clean default-bran Fleet supervision is an always-loaded operational contract; `docs/architecture.md`, `docs/turnend-guard.md`, the emitted session-start block, and script help own mechanisms and harness-specific recipes. -Whenever work is under way, keep exactly one live supervision cycle using the emitted protocol for this primary harness. -X mode may require that same live cycle with no fleet work. +Whenever work is under way or refill remains pending, keep exactly one live supervision cycle using the emitted protocol for this primary harness. +X mode or an unmet concurrency floor may require that same live cycle with no fleet metadata. Do not substitute another harness's wait shape, use shell `&`, or create a second cycle when a healthy one already exists. For every actionable wake, follow the ordinary-wake continuation in the emitted protocol; use its repair action only when the live cycle is missing or failed. No turn ends blind while work is under way, including turns described as holding or waiting. @@ -369,8 +370,8 @@ Handle actionable wakes as follows: 1. For `signal:`, read the listed event lines first, then reconcile current state only where action depends on it. 2. For `stale:`, inspect the recorded endpoint and load `stuck-crewmate-recovery` for a stopped, looping, confused, or unresponsive worker; a deep-inspection reason also requires current-state and validation-log inspection. 3. For `check:`, act on the named poll result, including merges, X-mode events, and refill signals. - A `check: refill completion :` or `check: refill floor:` wake is claim-next capacity, not a blind spawn: run the normal claim-and-dispatch procedure (verify-at-pickup, atomic claim, exclusions, held and parked items, date gates). - Until ship worktree leases land, hold pool dispatch when `fm_refill_has_parked_unpushed` reports parked unpushed commits (`bin/fm-refill-lib.sh`). + A `check: refill completion :`, `check: refill floor:`, or `check: refill pending:` wake is claim-next capacity, not a blind spawn: run the normal claim-and-dispatch procedure (verify-at-pickup, atomic claim, exclusions, held and parked items, date gates). + Until ship worktree leases land, hold pool dispatch unless `fm_refill_has_parked_unpushed` proves every non-`local-only` ship worktree readable and free of unpushed commits (`bin/fm-refill-lib.sh`). 4. For `heartbeat:`, review the whole fleet from the structured fleet view, reconcile suspicious tasks and PR state, update the backlog, and never report an unchanged fleet as progress. When any wake reports a merged PR for a project cloned in this home, refresh that clone through the guarded fleet-sync path. diff --git a/bin/fm-claude-stop-autoarm.sh b/bin/fm-claude-stop-autoarm.sh index df9ee1128f..26e40d7f02 100755 --- a/bin/fm-claude-stop-autoarm.sh +++ b/bin/fm-claude-stop-autoarm.sh @@ -18,8 +18,8 @@ # - AFK: while state/.afk exists the away daemon owns the watcher and triage; # this hook exits 0 and NEVER rewakes the primary (checked again at # translation time so a mid-cycle AFK transition is honored). -# - Need: arms only while work is in flight (state/*.meta) or X mode has a -# relay poll to run (state/x-watch.check.sh); an idle home exits 0. +# - Need: arms while work is in flight, X mode has a relay poll, or refill +# work remains queued, pending, or below a configured floor. # - Single-flight: Claude does not dedupe async hooks, so a home-scoped owner # lock (state/.claude-autoarm.lock) admits exactly one owner; every other # concurrent firing exits 0 without translating, which keeps one event @@ -89,7 +89,7 @@ fi # --- AFK: the away daemon owns the watcher and triage; never rewake ---------- [ -e "$STATE/.afk" ] && exit 0 -# --- need: in-flight work or an X-mode relay poll ---------------------------- +# --- need: in-flight work, X-mode relay polling, or refill work -------------- need_supervision() { fm_supervision_needed "$STATE" "$GRACE" } diff --git a/bin/fm-guard.sh b/bin/fm-guard.sh index e36b7f46b0..2bad904c8b 100755 --- a/bin/fm-guard.sh +++ b/bin/fm-guard.sh @@ -5,7 +5,7 @@ # First, always warn if the firstmate primary checkout (FM_ROOT) is on a named # non-default branch, because that means firstmate-on-itself work landed in the # primary instead of an isolated worktree. -# Then, if any task is in flight (a state/.meta exists) and the watcher's +# Then, if any task, X-mode relay, or refill work needs supervision and the watcher's # liveness beacon (state/.last-watcher-beat, touched every poll cycle) is # missing or older than FM_GUARD_GRACE seconds, prints a loud, clearly delimited # banner so the agent cannot skim past it in the tool output of whatever it was @@ -148,7 +148,7 @@ fm_supervision_status "$STATE" "$GRACE" in_flight=$FM_SUP_IN_FLIGHT watcher_fresh=$FM_SUP_WATCHER_FRESH beacon_desc=$FM_SUP_BEACON_DESC -if [ "$in_flight" -eq 0 ]; then +if [ "$FM_SUP_NEEDED" = false ]; then # Leave the unhealthy state (no work riding on the watcher): clear so a later # in-flight + stale combination is a fresh episode even if the beacon is still # absent with the same key string. @@ -187,7 +187,13 @@ if [ "$watcher_fresh" = false ]; then { printf '●%s\n' "$rule" printf '● WATCHER DOWN - SUPERVISION IS OFF\n' - printf '● %s task(s) in flight, but no watcher has a fresh beacon (last beat: %s, grace %ss).\n' "$in_flight" "$beacon_desc" "$GRACE" + if [ "$in_flight" -gt 0 ]; then + printf '● %s task(s) in flight, but no watcher has a fresh beacon (last beat: %s, grace %ss).\n' "$in_flight" "$beacon_desc" "$GRACE" + elif [ "$FM_SUP_REFILL_NEEDED" = true ]; then + printf '● Refill work is pending, but no watcher has a fresh beacon (last beat: %s, grace %ss).\n' "$beacon_desc" "$GRACE" + else + printf '● X-mode relay polling is active, but no watcher has a fresh beacon (last beat: %s, grace %ss).\n' "$beacon_desc" "$GRACE" + fi if [ "$READ_ONLY" -eq 1 ]; then printf '● This read-only session should report the lapse, not repair it.\n' else diff --git a/bin/fm-refill-lib.sh b/bin/fm-refill-lib.sh index c1fc630ffb..ee42a0fe70 100755 --- a/bin/fm-refill-lib.sh +++ b/bin/fm-refill-lib.sh @@ -1,53 +1,21 @@ #!/usr/bin/env bash -# Shared completion-triggered refill and concurrency-floor signalling. +# shellcheck disable=SC2034 +# Shared completion-triggered refill, concurrency-floor signalling, and refill +# supervision ownership. # -# OWNERSHIP: this library is the single owner of: -# - durable completion-refill wakes (kind=check, key=refill:) -# - concurrency-floor top-up wakes (kind=check, key=refill-floor) -# - per-task completion idempotence markers (state/.refill-completion-) -# - config/concurrency-floor parsing -# - the pre-lease parked-unpushed hazard probe used before top-up dispatch -# -# PROBLEM: continuous refill was a rule the supervising agent had to remember -# mid-turn. Long single turns that landed work never reached claim-next, so the -# queue starved. Completion events already existed (merge-poll wake, teardown -# backlog reminder) but neither emitted a durable signal the agent must act on. -# -# CONTRACT: -# - Emits durable wake-queue records only. Never spawns, claims, or edits the -# backlog. The supervising agent still runs the normal claim-and-dispatch -# procedure (verify-at-pickup, atomic claim, exclusions, held/parked). -# - Completion emit is idempotent per task id: a second delivery of the same -# completion event (merge re-fire, teardown retry, afk re-escalation) does -# not enqueue a second refill wake. -# - Floor emit is independent of completion: when live ship count is below -# config/concurrency-floor, one top-up wake is enqueued (wake-queue key -# dedupe collapses duplicates still pending). Absent/0/invalid config = off. -# - Until ship worktree leases land (fix-lease-ship-worktrees), dispatch must -# hold while any live ship worktree has unpushed commits; this library only -# probes that hazard and never reimplements pool leasing. -# -# Usage (source from bin/ scripts): -# . "$SCRIPT_DIR/fm-refill-lib.sh" -# fm_refill_emit_completion # 0 if newly emitted, 1 if deduped/skip -# fm_refill_emit_floor_if_needed # 0 if newly emitted, 1 if no top-up -# fm_refill_concurrency_floor # prints non-negative integer target -# fm_refill_live_ship_count # prints current live ship meta count -# fm_refill_has_parked_unpushed # 0 when hazard present, 1 when clear -# -# FM_REFILL_REASON is set to the wake payload when an emit returns 0. +# Completion and floor events only signal the normal claim-and-dispatch path. +# They never spawn, claim, or bypass pickup checks, exclusions, holds, or parks. +# A home remains supervision-active while a refill wake is queued, a completion +# or floor refill remains unhandled, a completion receipt is pending, or an +# enabled concurrency floor is below target. _FM_REFILL_LIB_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" # shellcheck source=bin/fm-wake-lib.sh . "$_FM_REFILL_LIB_DIR/fm-wake-lib.sh" -# Prefer the caller's FM_CONFIG_OVERRIDE; otherwise the home's config dir. -# wake-lib already resolved FM_HOME/STATE for this process. FM_REFILL_CONFIG="${FM_CONFIG_OVERRIDE:-$FM_HOME/config}" - -# Set to the wake payload when an emit returns 0; callers (fm-watch.sh) read it. -# shellcheck disable=SC2034 # External consumer reads this after a successful emit. FM_REFILL_REASON= +FM_REFILL_HAZARD= fm_refill_task_id_valid() { local id=$1 @@ -57,57 +25,72 @@ fm_refill_task_id_valid() { [ "${#id}" -le 64 ] } -# Print the configured concurrency floor (live ships target). Absent file, -# unreadable path, symlink, empty, zero, or non-digits → 0 (feature off). +fm_refill_normalize_count() { + local value=$1 + value=${value#"${value%%[!0]*}"} + [ -n "$value" ] || value=0 + printf '%s\n' "$value" +} + +fm_refill_count_below_target() { + local live floor + live=$(fm_refill_normalize_count "$1") + floor=$(fm_refill_normalize_count "$2") + [ "${#live}" -lt "${#floor}" ] && return 0 + [ "${#live}" -gt "${#floor}" ] && return 1 + [[ "$live" < "$floor" ]] +} + fm_refill_concurrency_floor() { - local f line - f="$FM_REFILL_CONFIG/concurrency-floor" + local config=${1:-$FM_REFILL_CONFIG} f line lines + f="$config/concurrency-floor" if [ ! -f "$f" ] || [ -L "$f" ]; then printf '0\n' return 0 fi - line=$(head -n 1 "$f" 2>/dev/null | tr -d '[:space:]') || line= + lines=$(awk 'END { print NR + 0 }' "$f" 2>/dev/null) || lines=0 + [ "$lines" = 1 ] || { printf '0\n'; return 0; } + line=$(head -n 1 "$f" 2>/dev/null) || line= + line=${line#"${line%%[![:space:]]*}"} + line=${line%"${line##*[![:space:]]}"} case "$line" in - ''|0|*[!0-9]*) - printf '0\n' - ;; - *) - printf '%s\n' "$line" - ;; + ''|0|*[!0-9]*) printf '0\n' ;; + *) fm_refill_normalize_count "$line" ;; esac } -# Count live ship workers from state/.meta. Missing kind defaults to ship -# (spawn's historical default). Scouts and secondmates do not count. fm_refill_live_ship_count() { - local meta id kind n=0 + local state=${1:-$STATE} meta id kind verdict n=0 + local reader=${FM_CREW_STATE_BIN:-$_FM_REFILL_LIB_DIR/fm-crew-state.sh} shopt -s nullglob - for meta in "$STATE"/*.meta; do + for meta in "$state"/*.meta; do [ -f "$meta" ] && [ ! -L "$meta" ] || continue id=$(basename "$meta" .meta) fm_refill_task_id_valid "$id" || continue kind=$(grep '^kind=' "$meta" 2>/dev/null | tail -1 | cut -d= -f2- || true) [ -n "$kind" ] || kind=ship - case "$kind" in - ship) n=$((n + 1)) ;; + [ "$kind" = ship ] || continue + verdict=$(FM_HOME="$(dirname "$state")" FM_STATE_OVERRIDE="$state" \ + FM_CREW_STATE_NM_TIMEOUT="${FM_REFILL_CREW_STATE_NM_TIMEOUT:-2}" \ + "$reader" "$id" 2>/dev/null) || verdict= + case "$verdict" in + 'state: working · '*) n=$((n + 1)) ;; esac done shopt -u nullglob printf '%s\n' "$n" } -# 0 when any live ship worktree still has commits not on any remote (the -# pre-lease pool-recycle hazard). local-only ships are excluded: they land via -# local main rather than a remote push. Returns 1 when clear or unreadable. -# Coordinate with fix-lease-ship-worktrees: once ship worktrees are leased, the -# supervising agent can stop treating this probe as a hard hold. fm_refill_has_parked_unpushed() { - local meta id kind mode wt unpushed + local state=${1:-$STATE} meta kind mode wt unpushed + FM_REFILL_HAZARD= shopt -s nullglob - for meta in "$STATE"/*.meta; do - [ -f "$meta" ] && [ ! -L "$meta" ] || continue - id=$(basename "$meta" .meta) - fm_refill_task_id_valid "$id" || continue + for meta in "$state"/*.meta; do + if [ ! -f "$meta" ] || [ -L "$meta" ]; then + FM_REFILL_HAZARD="unreadable metadata state for $(basename "$meta" .meta)" + shopt -u nullglob + return 0 + fi kind=$(grep '^kind=' "$meta" 2>/dev/null | tail -1 | cut -d= -f2- || true) [ -n "$kind" ] || kind=ship [ "$kind" = ship ] || continue @@ -115,9 +98,18 @@ fm_refill_has_parked_unpushed() { [ -n "$mode" ] || mode=no-mistakes [ "$mode" != local-only ] || continue wt=$(grep '^worktree=' "$meta" 2>/dev/null | tail -1 | cut -d= -f2- || true) - [ -n "$wt" ] && [ -d "$wt" ] || continue - unpushed=$(git -C "$wt" log --oneline HEAD --not --remotes -- 2>/dev/null) || continue + if [ -z "$wt" ] || [ ! -d "$wt" ]; then + FM_REFILL_HAZARD="unknown worktree state for $(basename "$meta" .meta)" + shopt -u nullglob + return 0 + fi + if ! unpushed=$(git -C "$wt" log --oneline HEAD --not --remotes -- 2>/dev/null); then + FM_REFILL_HAZARD="unreadable git state for $(basename "$meta" .meta)" + shopt -u nullglob + return 0 + fi if [ -n "$unpushed" ]; then + FM_REFILL_HAZARD="unpushed commits in $(basename "$meta" .meta)" shopt -u nullglob return 0 fi @@ -126,81 +118,236 @@ fm_refill_has_parked_unpushed() { return 1 } +fm_refill_hold_suffix() { + local state=${1:-$STATE} + if fm_refill_has_parked_unpushed "$state"; then + printf '%s' "; HOLD: parked ship worktree safety is not proven ($FM_REFILL_HAZARD) - do not pool-dispatch until leases protect ship worktrees or the state is proven clear" + fi +} + fm_refill_completion_marker() { - local id=$1 - printf '%s/.refill-completion-%s\n' "$STATE" "$id" + printf '%s/.refill-completion-%s\n' "${2:-$STATE}" "$1" } -# Claim exclusive right to emit one completion refill for . -# Uses noclobber create so concurrent and re-fired deliveries race safely. -# Returns 0 on fresh claim, 1 if already claimed or id invalid. -fm_refill_claim_completion() { - local id=$1 marker - fm_refill_task_id_valid "$id" || return 1 - marker=$(fm_refill_completion_marker "$id") - mkdir -p "$STATE" || return 1 - if ( - set -C - umask 077 - : >"$marker" - ) 2>/dev/null; then - printf '%s\n' "$(date +%s)" >"$marker" || true - return 0 +fm_refill_completion_needed_marker() { + printf '%s/.refill-needed-completion\n' "${1:-$STATE}" +} + +fm_refill_floor_needed_marker() { + printf '%s/.refill-needed-floor\n' "${1:-$STATE}" +} + +fm_refill_atomic_write() { + local path=$1 value=$2 dir tmp + dir=$(dirname "$path") + mkdir -p "$dir" || return 1 + tmp=$(mktemp "$dir/.refill-write.XXXXXX") || return 1 + if ! (umask 077; printf '%s\n' "$value" > "$tmp") || ! mv -f "$tmp" "$path"; then + rm -f "$tmp" 2>/dev/null || true + return 1 fi - return 1 } -fm_refill_release_completion_claim() { - local id=$1 marker - fm_refill_task_id_valid "$id" || return 0 - marker=$(fm_refill_completion_marker "$id") - rm -f -- "$marker" 2>/dev/null || true +fm_refill_mark_needed() { + fm_refill_atomic_write "$1" "needed $(date +%s)" +} + +fm_refill_marker_state() { + local marker=$1 value + [ -e "$marker" ] || { printf 'absent\n'; return 0; } + [ -f "$marker" ] && [ ! -L "$marker" ] || { printf 'pending\n'; return 0; } + value=$(head -n 1 "$marker" 2>/dev/null || true) + case "$value" in + pending*) printf 'pending\n' ;; + committed*) printf 'committed\n' ;; + ''|*[!0-9]*) printf 'pending\n' ;; + *) printf 'committed\n' ;; + esac +} + +fm_refill_queue_payload_locked() { + local key=$1 queue=${2:-$FM_WAKE_QUEUE} + awk -F '\t' -v key="$key" '$3 == "check" && $4 == key { payload = $5 } END { if (payload != "") print payload; else exit 1 }' \ + "$queue" 2>/dev/null +} + +fm_refill_queue_any_payload_locked() { + local queue=${1:-$FM_WAKE_QUEUE} + awk -F '\t' '$3 == "check" && $4 ~ /^refill([:-]|$)/ { payload = $5 } END { if (payload != "") print payload; else exit 1 }' \ + "$queue" 2>/dev/null +} + +fm_refill_commit_completion_marker() { + local id=$1 state=${2:-$STATE} + fm_refill_atomic_write "$(fm_refill_completion_marker "$id" "$state")" "committed $(date +%s)" +} + +fm_refill_finalize_completion_receipts() { + local queue=${1:-$FM_WAKE_QUEUE} state=${2:-$STATE} key id + while IFS= read -r key; do + [ -n "$key" ] || continue + id=${key#refill:} + fm_refill_task_id_valid "$id" || return 1 + fm_refill_commit_completion_marker "$id" "$state" || return 1 + fm_refill_mark_needed "$(fm_refill_completion_needed_marker "$state")" || return 1 + done < <(awk -F '\t' '$3 == "check" && $4 ~ /^refill:/ && !seen[$4]++ { print $4 }' "$queue" 2>/dev/null) +} + +fm_refill_recover_pending_completions() { + local state=${1:-$STATE} marker id + shopt -s nullglob + for marker in "$state"/.refill-completion-*; do + [ "$(fm_refill_marker_state "$marker")" = pending ] || continue + id=${marker##*/.refill-completion-} + fm_refill_task_id_valid "$id" || continue + fm_refill_emit_completion "$id" >/dev/null 2>&1 || true + done + shopt -u nullglob +} + +fm_refill_emit_floor_if_needed() { + local state=${1:-$STATE} config=${2:-$FM_REFILL_CONFIG} floor live payload existing status=1 + FM_REFILL_REASON= + floor=$(fm_refill_concurrency_floor "$config") + case "$floor" in + ''|0|*[!0-9]*) + fm_lock_acquire_wait "$FM_WAKE_QUEUE_LOCK" + rm -f "$(fm_refill_floor_needed_marker "$state")" 2>/dev/null || true + fm_lock_release "$FM_WAKE_QUEUE_LOCK" + return 1 + ;; + esac + live=$(fm_refill_live_ship_count "$state") + case "$live" in ''|*[!0-9]*) live=0 ;; esac + if ! fm_refill_count_below_target "$live" "$floor"; then + fm_lock_acquire_wait "$FM_WAKE_QUEUE_LOCK" + rm -f "$(fm_refill_floor_needed_marker "$state")" 2>/dev/null || true + fm_lock_release "$FM_WAKE_QUEUE_LOCK" + return 1 + fi + payload="check: refill floor: live ships $live below target $floor; run normal claim-and-dispatch (tasks-axi ready, date gates, exclusions, held and parked); do not blind-spawn$(fm_refill_hold_suffix "$state")" + fm_lock_acquire_wait "$FM_WAKE_QUEUE_LOCK" + fm_refill_mark_needed "$(fm_refill_floor_needed_marker "$state")" || { + fm_lock_release "$FM_WAKE_QUEUE_LOCK" + return 1 + } + existing=$(fm_refill_queue_payload_locked refill-floor 2>/dev/null || true) + if [ -n "$existing" ]; then + FM_REFILL_REASON=$existing + elif fm_wake_append_locked check refill-floor "$payload"; then + FM_REFILL_REASON=$payload + status=0 + fi + fm_lock_release "$FM_WAKE_QUEUE_LOCK" + return "$status" } -# Emit one durable completion-refill wake for . -# Idempotent: a second call for the same id is a no-op (return 1). -# Does not spawn. On success sets FM_REFILL_REASON and returns 0. fm_refill_emit_completion() { - local id=$1 key payload - # shellcheck disable=SC2034 # Cleared for callers that read FM_REFILL_REASON after emit. + local id=$1 marker key payload existing marker_state status=1 FM_REFILL_REASON= fm_refill_task_id_valid "$id" || return 1 - fm_refill_claim_completion "$id" || return 1 + marker=$(fm_refill_completion_marker "$id") key="refill:$id" - payload="check: refill completion $id: run normal claim-and-dispatch (tasks-axi ready, date gates, exclusions, held and parked); do not blind-spawn" - if ! fm_wake_append check "$key" "$payload"; then - fm_refill_release_completion_claim "$id" + payload="check: refill completion $id: run normal claim-and-dispatch (tasks-axi ready, date gates, exclusions, held and parked); do not blind-spawn$(fm_refill_hold_suffix "$STATE")" + fm_lock_acquire_wait "$FM_WAKE_QUEUE_LOCK" + marker_state=$(fm_refill_marker_state "$marker") + if [ "$marker_state" = committed ]; then + fm_lock_release "$FM_WAKE_QUEUE_LOCK" + fm_refill_emit_floor_if_needed >/dev/null 2>&1 || true + return 1 + fi + if [ "$marker_state" = absent ]; then + fm_refill_atomic_write "$marker" "pending $(date +%s)" || { + fm_lock_release "$FM_WAKE_QUEUE_LOCK" + return 1 + } + fi + fm_refill_mark_needed "$(fm_refill_completion_needed_marker)" || { + fm_lock_release "$FM_WAKE_QUEUE_LOCK" return 1 + } + if [ "${FM_REFILL_TEST_STOP_AFTER_PENDING:-0}" = 1 ]; then + fm_lock_release "$FM_WAKE_QUEUE_LOCK" + return 75 fi - # shellcheck disable=SC2034 # Public result consumed by fm-watch.sh after a successful emit. - FM_REFILL_REASON=$payload - return 0 + existing=$(fm_refill_queue_payload_locked "$key" 2>/dev/null || true) + if [ -n "$existing" ]; then + fm_refill_commit_completion_marker "$id" || { + fm_lock_release "$FM_WAKE_QUEUE_LOCK" + return 1 + } + FM_REFILL_REASON=$existing + elif fm_wake_append_locked check "$key" "$payload"; then + fm_refill_commit_completion_marker "$id" || { + fm_lock_release "$FM_WAKE_QUEUE_LOCK" + return 1 + } + FM_REFILL_REASON=$payload + status=0 + fi + fm_lock_release "$FM_WAKE_QUEUE_LOCK" + fm_refill_emit_floor_if_needed >/dev/null 2>&1 || true + [ "$status" -ne 0 ] || FM_REFILL_REASON=$payload + return "$status" } -# Emit one durable floor top-up wake when live ships are below the configured -# target. Absent/zero floor → no-op. At or above target → no-op. Pending queue -# key refill-floor collapses duplicates until drained; a later drop below the -# floor after drain can emit again (independent safety net). -# Returns 0 when newly enqueued, 1 when no top-up is needed or enqueue fails. -fm_refill_emit_floor_if_needed() { - local floor live payload - # shellcheck disable=SC2034 # Cleared for callers that read FM_REFILL_REASON after emit. +fm_refill_pending_marker_exists() { + local state=${1:-$STATE} marker id + [ -f "$(fm_refill_completion_needed_marker "$state")" ] && return 0 + [ -f "$(fm_refill_floor_needed_marker "$state")" ] && return 0 + shopt -s nullglob + for marker in "$state"/.refill-completion-*; do + [ "$(fm_refill_marker_state "$marker")" = pending ] || continue + id=${marker##*/.refill-completion-} + fm_refill_task_id_valid "$id" || continue + shopt -u nullglob + return 0 + done + shopt -u nullglob + return 1 +} + +fm_refill_supervision_needed() { + local state=${1:-$STATE} config=${2:-$FM_REFILL_CONFIG} floor live + fm_refill_queue_any_payload_locked "$state/.wake-queue" >/dev/null 2>&1 && return 0 + fm_refill_pending_marker_exists "$state" && return 0 + floor=$(fm_refill_concurrency_floor "$config") + case "$floor" in ''|0|*[!0-9]*) return 1 ;; esac + live=$(fm_refill_live_ship_count "$state") + case "$live" in ''|*[!0-9]*) live=0 ;; esac + fm_refill_count_below_target "$live" "$floor" +} + +fm_refill_emit_pending_if_needed() { + local payload existing FM_REFILL_REASON= - floor=$(fm_refill_concurrency_floor) - case "$floor" in - ''|0|*[!0-9]*) return 1 ;; - esac - live=$(fm_refill_live_ship_count) - case "$live" in - ''|*[!0-9]*) live=0 ;; - esac - [ "$live" -lt "$floor" ] || return 1 - payload="check: refill floor: live ships $live below target $floor; run normal claim-and-dispatch (tasks-axi ready, date gates, exclusions, held and parked); do not blind-spawn" - if fm_refill_has_parked_unpushed; then - payload="$payload; HOLD: parked ship worktree has unpushed commits - do not pool-dispatch until leases protect ship worktrees or unpushed work is cleared" + fm_refill_emit_floor_if_needed >/dev/null 2>&1 || true + fm_refill_recover_pending_completions "$STATE" + fm_lock_acquire_wait "$FM_WAKE_QUEUE_LOCK" + existing=$(fm_refill_queue_any_payload_locked 2>/dev/null || true) + if [ -n "$existing" ]; then + FM_REFILL_REASON=$existing + fm_lock_release "$FM_WAKE_QUEUE_LOCK" + return 0 + fi + if ! fm_refill_pending_marker_exists "$STATE"; then + fm_lock_release "$FM_WAKE_QUEUE_LOCK" + return 1 fi - fm_wake_append check refill-floor "$payload" || return 1 - # shellcheck disable=SC2034 # Public result consumed by fm-watch.sh after a successful emit. - FM_REFILL_REASON=$payload - return 0 + payload="check: refill pending: run normal claim-and-dispatch (tasks-axi ready, date gates, exclusions, held and parked); do not blind-spawn$(fm_refill_hold_suffix "$STATE")" + if fm_wake_append_locked check refill-pending "$payload"; then + FM_REFILL_REASON=$payload + fm_lock_release "$FM_WAKE_QUEUE_LOCK" + return 0 + fi + fm_lock_release "$FM_WAKE_QUEUE_LOCK" + return 1 +} + +fm_refill_dispatch_cycle_completed() { + local state=${1:-$STATE} config=${2:-$FM_REFILL_CONFIG} + fm_lock_acquire_wait "$FM_WAKE_QUEUE_LOCK" + rm -f "$(fm_refill_completion_needed_marker "$state")" 2>/dev/null || true + fm_lock_release "$FM_WAKE_QUEUE_LOCK" + fm_refill_emit_floor_if_needed "$state" "$config" >/dev/null 2>&1 || true } diff --git a/bin/fm-session-start.sh b/bin/fm-session-start.sh index e4f21d0358..6df872666b 100755 --- a/bin/fm-session-start.sh +++ b/bin/fm-session-start.sh @@ -299,6 +299,9 @@ if [ "$READ_ONLY" -eq 1 ]; then GUARD_OUT=$(FM_GUARD_READ_ONLY=1 "$SCRIPT_DIR/fm-guard.sh" 2>&1) [ -n "$GUARD_OUT" ] && printf '%s\n' "$GUARD_OUT" else + # shellcheck source=bin/fm-refill-lib.sh + . "$SCRIPT_DIR/fm-refill-lib.sh" + fm_refill_emit_pending_if_needed >/dev/null 2>&1 || true DRAIN_OUT=$("$SCRIPT_DIR/fm-wake-drain.sh" 2>&1) if [ -n "$DRAIN_OUT" ]; then printf '%s\n' "$DRAIN_OUT" diff --git a/bin/fm-spawn.sh b/bin/fm-spawn.sh index b6ffdfd347..81c8d767d6 100755 --- a/bin/fm-spawn.sh +++ b/bin/fm-spawn.sh @@ -167,6 +167,8 @@ SUB_HOME_MARKER=".fm-secondmate-home" . "$SCRIPT_DIR/fm-ff-lib.sh" # shellcheck source=bin/fm-wake-lib.sh . "$SCRIPT_DIR/fm-wake-lib.sh" +# shellcheck source=bin/fm-refill-lib.sh +. "$SCRIPT_DIR/fm-refill-lib.sh" # shellcheck source=bin/fm-config-inherit-lib.sh . "$SCRIPT_DIR/fm-config-inherit-lib.sh" # shellcheck source=bin/fm-backend.sh @@ -1722,4 +1724,5 @@ if [ "$KIND" = secondmate ]; then fi fi +[ "$KIND" != ship ] || fm_refill_dispatch_cycle_completed "$STATE" "$CONFIG" echo "spawned $ID harness=$HARNESS kind=$KIND mode=$MODE yolo=$YOLO window=$META_WINDOW worktree=$WT" diff --git a/bin/fm-supervision-lib.sh b/bin/fm-supervision-lib.sh index 1930700d2a..df83c0bec1 100644 --- a/bin/fm-supervision-lib.sh +++ b/bin/fm-supervision-lib.sh @@ -3,14 +3,17 @@ # Usage: . bin/fm-supervision-lib.sh # # Reports whether a firstmate home needs supervision because it has in-flight -# work (a state/.meta exists) or an X-mode relay poll -# (state/x-watch.check.sh), and whether its watcher has a fresh liveness beacon +# work, an X-mode relay poll, or refill work, and whether its watcher has a fresh liveness beacon # (state/.last-watcher-beat, touched every poll cycle, within the grace window). # bin/fm-guard.sh keeps its task-specific grace-based warning predicate; # bin/fm-turnend-guard.sh uses the status fields here for its banner but performs # its end-of-turn block decision with the live watcher lock check in # bin/fm-wake-lib.sh. +_FM_SUP_LIB_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +# shellcheck source=bin/fm-refill-lib.sh +. "$_FM_SUP_LIB_DIR/fm-refill-lib.sh" + # Portable mtime; Linux stat lacks -f, macOS stat lacks -c. fm_sup_stat_mtime() { if [ "$(uname)" = Darwin ]; then @@ -23,10 +26,11 @@ fm_sup_stat_mtime() { # fm_supervision_status [grace-seconds] # Populates, for the state dir at $1: # FM_SUP_IN_FLIGHT count of state/*.meta (in-flight tasks) -# FM_SUP_NEEDED true/false - in-flight work or an X-mode relay poll +# FM_SUP_NEEDED true/false - in-flight work, X-mode relay, or refill work # FM_SUP_WATCHER_FRESH true/false - a watcher beacon within the grace window # FM_SUP_BEACON_DESC human-readable beacon age, for banners ("never" if absent) # FM_SUP_QUEUE_PENDING true/false - state/.wake-queue has unread records +# FM_SUP_REFILL_NEEDED true/false - refill queue, marker, or floor needs a cycle # grace-seconds defaults to $FM_GUARD_GRACE, then 300, matching fm-guard.sh. # Always returns 0; callers read the vars, or use fm_supervision_unhealthy below. fm_supervision_status() { @@ -36,12 +40,16 @@ fm_supervision_status() { FM_SUP_WATCHER_FRESH=false FM_SUP_BEACON_DESC=never FM_SUP_QUEUE_PENDING=false + FM_SUP_REFILL_NEEDED=false for meta in "$state"/*.meta; do [ -e "$meta" ] || continue FM_SUP_IN_FLIGHT=$((FM_SUP_IN_FLIGHT + 1)) done - if [ "$FM_SUP_IN_FLIGHT" -gt 0 ] || [ -f "$state/x-watch.check.sh" ]; then + if fm_refill_supervision_needed "$state" "${FM_CONFIG_OVERRIDE:-$(dirname "$state")/config}"; then + FM_SUP_REFILL_NEEDED=true + fi + if [ "$FM_SUP_IN_FLIGHT" -gt 0 ] || [ -f "$state/x-watch.check.sh" ] || [ "$FM_SUP_REFILL_NEEDED" = true ]; then FM_SUP_NEEDED=true fi @@ -64,7 +72,7 @@ fm_supervision_status() { } # fm_supervision_needed [grace-seconds] -# Exit 0 (true) exactly when in-flight work or an X-mode relay poll needs a +# Exit 0 (true) exactly when in-flight work, an X-mode relay poll, or refill work needs a # watcher. Exit 1 (false) for an idle home. fm_supervision_needed() { fm_supervision_status "$@" diff --git a/bin/fm-turnend-guard.sh b/bin/fm-turnend-guard.sh index 2e96fb33e4..531f16d72f 100755 --- a/bin/fm-turnend-guard.sh +++ b/bin/fm-turnend-guard.sh @@ -134,16 +134,9 @@ budget_reset() { } fm_supervision_status "$STATE" "$GRACE" -if [ "$CLAUDE_MODE" -eq 1 ]; then - if [ "$FM_SUP_NEEDED" = false ]; then - budget_reset - exit 0 - fi -else - if [ "$FM_SUP_IN_FLIGHT" -eq 0 ]; then - budget_reset - exit 0 - fi +if [ "$FM_SUP_NEEDED" = false ]; then + budget_reset + exit 0 fi if fm_watcher_healthy "$STATE" "$WATCH" "$GRACE" "$FM_HOME"; then budget_reset @@ -164,6 +157,8 @@ block_stop() { printf '● TURN WOULD END BLIND - SUPERVISION IS OFF\n' if [ "$FM_SUP_IN_FLIGHT" -gt 0 ]; then printf '● %s task(s) in flight, but no live watcher holds this home lock (last beat: %s).\n' "$FM_SUP_IN_FLIGHT" "$FM_SUP_BEACON_DESC" + elif [ "$FM_SUP_REFILL_NEEDED" = true ]; then + printf '● Refill work needs supervision, but no live watcher holds this home lock (last beat: %s).\n' "$FM_SUP_BEACON_DESC" else printf '● X-mode relay polling needs supervision, but no live watcher holds this home lock (last beat: %s).\n' "$FM_SUP_BEACON_DESC" fi @@ -229,6 +224,8 @@ if [ "$COUNT" -gt "$BLOCK_BUDGET" ]; then budget_reset if [ "$FM_SUP_IN_FLIGHT" -gt 0 ]; then NEED_DESC="$FM_SUP_IN_FLIGHT task(s) in flight" + elif [ "$FM_SUP_REFILL_NEEDED" = true ]; then + NEED_DESC="refill work pending" else NEED_DESC="X-mode relay polling active" fi diff --git a/bin/fm-wake-drain.sh b/bin/fm-wake-drain.sh index c3bf7335c0..2f0f86b71a 100755 --- a/bin/fm-wake-drain.sh +++ b/bin/fm-wake-drain.sh @@ -6,6 +6,8 @@ set -u SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" # shellcheck source=bin/fm-wake-lib.sh . "$SCRIPT_DIR/fm-wake-lib.sh" +# shellcheck source=bin/fm-refill-lib.sh +. "$SCRIPT_DIR/fm-refill-lib.sh" DRAIN_TMP= DRAIN_LOCK_HELD=false @@ -51,6 +53,8 @@ if [ ! -s "$FM_WAKE_QUEUE" ]; then exit 0 fi +fm_refill_finalize_completion_receipts "$FM_WAKE_QUEUE" "$STATE" || exit 1 + DRAIN_TMP="$STATE/.wake-queue.drain.$(fm_current_pid)" rm -f "$DRAIN_TMP" mv "$FM_WAKE_QUEUE" "$DRAIN_TMP" || exit 1 diff --git a/bin/fm-wake-lib.sh b/bin/fm-wake-lib.sh index 8cec58bec1..4a9e047930 100755 --- a/bin/fm-wake-lib.sh +++ b/bin/fm-wake-lib.sh @@ -379,7 +379,7 @@ fm_wake_clean_field() { LC_ALL=C tr '\t\r\n' ' ' } -fm_wake_append() { +fm_wake_append_locked() { local kind=$1 key=$2 payload=$3 clean_key clean_payload epoch seq seq_file status case "$kind" in signal|stale|check|heartbeat) ;; @@ -392,7 +392,6 @@ fm_wake_append() { seq_file="$STATE/.wake-queue.seq" status=0 - fm_lock_acquire_wait "$FM_WAKE_QUEUE_LOCK" seq=$(cat "$seq_file" 2>/dev/null || echo 0) case "$seq" in ''|*[!0-9]*) seq=0 ;; @@ -402,6 +401,14 @@ fm_wake_append() { if [ "$status" -eq 0 ]; then printf '%s\t%s\t%s\t%s\t%s\n' "$epoch" "$seq" "$kind" "$clean_key" "$clean_payload" >> "$FM_WAKE_QUEUE" || status=$? fi + return "$status" +} + +fm_wake_append() { + local status + fm_lock_acquire_wait "$FM_WAKE_QUEUE_LOCK" + fm_wake_append_locked "$@" + status=$? fm_lock_release "$FM_WAKE_QUEUE_LOCK" return "$status" } diff --git a/bin/fm-watch.sh b/bin/fm-watch.sh index 3e2d270f86..3a07849ecc 100755 --- a/bin/fm-watch.sh +++ b/bin/fm-watch.sh @@ -50,6 +50,8 @@ # durable completion-triggered claim-next (fm-refill-lib) # check: refill floor: ... # concurrency-floor top-up when live ships are below target +# check: refill pending: ... +# durable refill need re-surfaced after its earlier wake drained # heartbeat fleet-scan backstop found an unsurfaced captain-relevant # status, unless afk is active # For normal supervision, resume the session-start primary-harness protocol @@ -712,6 +714,10 @@ if ! fm_pr_poll_retirement_recover_all "$STATE" "$SCRIPT_DIR/fm-pr-poll.sh"; the wake "$reason" fi +if fm_refill_emit_pending_if_needed; then + wake "$FM_REFILL_REASON" +fi + while :; do # Self-eviction: if the singleton lock no longer names this process, a second # watcher has taken over (e.g. a transient duplicate from a racy arm). Stand diff --git a/docs/architecture.md b/docs/architecture.md index 2ba24e8ac4..0e97b3636c 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -14,8 +14,12 @@ Repeated provably-working stale escalations on the same unchanged pane add an es A busy pane is otherwise exempt from staleness, but only until its latest `state/.turn-ended` marker reaches `FM_BUSY_TURN_MAX_SECS`, or its `state/.meta` spawn record reaches that age before any turn completes; past that bound it is routed through the same wedge escalation, with the identical reason, escalation count, and `demand-deep-inspection` marker, for inspection only - never an automatic interrupt, signal, or restart. Those actionable wakes are written to a durable local queue (`state/.wake-queue`) before detector state advances, so a missed process exit can be recovered by draining the queue. Completion also enqueues a durable refill wake through `bin/fm-refill-lib.sh`. -Merge-poll `merged` and successful ship/scout teardown each claim one per-task completion marker and append `check: refill completion :`. -An optional `config/concurrency-floor` target independently appends `check: refill floor:` when live ships fall below that count on the heartbeat cadence. +Merge-poll `merged` and successful ship/scout teardown each use one crash-recoverable per-task receipt and append `check: refill completion :` exactly once after a successful commit. +Queue drain finalizes any pending completion receipt before consuming its row, so a crash between receipt claim and append retries the missing wake while a committed wake remains deduped across restart. +An optional `config/concurrency-floor` target independently appends `check: refill floor:` when authoritative current-state reads show fewer working ships than the target. +Session start, watcher startup, completion, and heartbeat evaluate or surface refill state, while the shared supervision predicate keeps queued, pending, or below-target homes active with zero task metadata. +Successful ship dispatch clears a handled completion need, and meeting or disabling the floor clears its independent need. +Until ship worktree leases protect pool slots, unreadable or unproven non-local ship worktree state fails closed with a dispatch hold in every refill payload. Both signals only ask the supervising agent to run the ordinary claim-and-dispatch path; they never spawn, claim, or bypass exclusions. When a canonical validated PR poll returns exactly `merged`, the watcher appends that durable notification before publishing a private receipt bound to the poll's registration, bytes, file identities, metadata, provider, URL, and task ID. The receipt makes retirement safely retryable across restarts: fixed-path recovery revalidates the same evidence, removes the runnable check first, removes its registration and data sidecars, removes the receipt last, and preserves task metadata including `pr=` and `pr_head=`. diff --git a/docs/configuration.md b/docs/configuration.md index 469aff9893..a7c1e06ddc 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -28,11 +28,16 @@ Ordinary dead-direct-report recovery is owned by `stuck-crewmate-recovery`, whil `config/concurrency-floor` is an optional local, gitignored file under the effective Firstmate home that sets a target number of concurrent live ship workers for that home. The file is one non-negative base-10 integer on a single line; only a positive integer enables the floor. An absent, unreadable, symlinked, empty, zero, or non-numeric value means the floor is off. -When the floor is on and the count of live `kind=ship` metadata records in `state/` drops below the target, `bin/fm-refill-lib.sh` enqueues one durable `check: refill floor:` wake so the supervising agent runs the normal claim-and-dispatch procedure. +When the floor is on, `bin/fm-refill-lib.sh` counts only `kind=ship` workers whose authoritative `bin/fm-crew-state.sh` current state is `working`. +Dead, unknown, terminal, parked, paused, scout, and secondmate records do not count as live capacity. +When that live count drops below the target, the library enqueues one durable `check: refill floor:` wake so the supervising agent runs the normal claim-and-dispatch procedure. +The floor is evaluated on locked session start, completion, watcher startup, and watcher heartbeat. The floor is independent of completion-triggered refill (merge-poll and teardown). Both paths exist so a missed rule cannot starve the queue. Neither path spawns work: they only surface the signal. -Until ship worktree leases protect pool slots, a floor or completion refill payload may include a hold when any live non-`local-only` ship worktree still has unpushed commits. +Queued refill wakes, pending completion receipts, durable refill-needed markers, and an enabled below-target floor each keep the home supervision-active even when no task metadata remains. +A successful ship dispatch clears the handled completion need, while a met or disabled floor clears the floor need. +Until ship worktree leases protect pool slots, every refill payload includes a hold unless each non-`local-only` ship worktree can be read successfully and proven free of unpushed commits. Coordinate with that lease work rather than reimplementing pool protection here. This preference is per home and is not part of secondmate inherited configuration. diff --git a/docs/scripts.md b/docs/scripts.md index 1319235794..b80be04ec0 100644 --- a/docs/scripts.md +++ b/docs/scripts.md @@ -74,7 +74,7 @@ The shared no-mistakes gate refusal for fleet lifecycle entrypoints is summarize | `fm-vendor-auth-probe.sh`| Run one hard-bounded, non-destructive authentication probe of a named vendor CLI and report the fact | | `fm-wake-drain.sh` | Atomically drain queued watcher wakes, emit bounded best-effort status-event annotations, then assert watcher liveness | | `fm-wake-lib.sh` | Shared durable wake queue, portable locks, and watcher identity/health helpers | -| `fm-refill-lib.sh` | Completion-triggered refill and concurrency-floor top-up wakes (idempotent, no spawn) | +| `fm-refill-lib.sh` | Crash-recoverable completion refill, current-state concurrency floor, and refill supervision ownership | | `fm-classify-lib.sh` | Shared captain-relevant and declared-external-wait wake classification vocabulary | | `fm-send.sh` | Send one verified literal line or supported key through the target's recorded backend | | `fm-busy-lib.sh` | Single owner of the semantic busy-state contract: verdicts, source attribution, and per-harness sources | diff --git a/docs/turnend-guard.md b/docs/turnend-guard.md index 8ee750de39..14a4bf1b9a 100644 --- a/docs/turnend-guard.md +++ b/docs/turnend-guard.md @@ -25,9 +25,9 @@ An unmarked checkout or invalid marker falls through to the git-dir check. That check keeps crewmate and scout linked worktrees inert because their git dir differs from their git common dir. It also requires `AGENTS.md`, `bin/`, and the effective state directory. -For an in-scope primary, the guard counts in-flight work from `state/*.meta`. -The default cross-harness mode exits silently with no work in flight. -Claude's `--claude` mode also treats `state/x-watch.check.sh` as supervision need, so X-mode relay polling remains guarded without an in-flight task. +For an in-scope primary, the guard counts in-flight work from `state/*.meta` and also reads the shared refill supervision predicate. +The default cross-harness mode exits silently when the home has no supervision need. +Every supported primary guard treats `state/x-watch.check.sh` and queued, pending, or below-target refill work as supervision needs without requiring an in-flight task. Otherwise it calls `fm_watcher_healthy [grace-seconds] [home]` from `bin/fm-wake-lib.sh`, the same identity-matched lock and fresh-beacon check used by `bin/fm-watch-arm.sh`. A stale beacon blocks even when a watcher pid is live. A fresh leftover beacon blocks when the lock is missing, dead, or identity-mismatched. @@ -76,7 +76,7 @@ That warning uses `bin/fm-supervision-instructions.sh --repair-line`, so it alwa ## Compatibility limits - Child crewmate and scout worktrees are outside scope. -- A valid secondmate home is in scope; an idle secondmate endpoint with no X-mode relay poll remains healthy because it has no supervision need. +- A valid secondmate home is in scope; an idle secondmate endpoint with no X-mode relay poll or refill need remains healthy because it has no supervision need. - The direct-blocking and bounded passive-follow-up split is limited to the primary integrations listed above. - OpenCode headless mode and untrusted Grok project hooks remain fail-open at the host boundary. - Kimi Code CLI 0.29.1 exposes only global `[[hooks]]` configuration in `~/.kimi-code/config.toml`, including a `Stop` event with snake_case payload fields `hook_event_name`, `session_id`, `cwd`, and `stop_hook_active`. diff --git a/tests/fm-backend.test.sh b/tests/fm-backend.test.sh index 3052ebc422..483e8466b1 100755 --- a/tests/fm-backend.test.sh +++ b/tests/fm-backend.test.sh @@ -143,7 +143,7 @@ resolve_permissive_tmux_kill_ref() { # when this branch changes tmux dispatch semantics. OLD_BIN_UNCHANGED_SIBLINGS="fm-gate-refuse-lib.sh fm-guard.sh fm-lock-lib.sh fm-tasks-axi-lib.sh fm-pr-lib.sh fm-tangle-lib.sh fm-tmux-lib.sh fm-composer-lib.sh fm-wake-lib.sh fm-classify-lib.sh fm-supervision-lib.sh fm-ff-lib.sh fm-config-inherit-lib.sh fm-project-mode.sh fm-harness.sh fm-crew-state.sh fm-decision-hold.sh fm-backend.sh fm-operational-input.sh fm-public-followup-lib.sh fm-secondmate-registry-lib.sh fm-x-lib.sh" # A pull-request merge may add a new main-only dependency that the branch's older baseline does not have yet. -OLD_BIN_OPTIONAL_SIBLINGS="fm-pending-reply-lib.sh" +OLD_BIN_OPTIONAL_SIBLINGS="fm-pending-reply-lib.sh fm-refill-lib.sh" OLD_BIN_REFACTORED="fm-send.sh fm-peek.sh fm-watch.sh fm-spawn.sh fm-teardown.sh fm-marker-lib.sh" build_old_bin() { # -> echoes root dir (root/bin/