Skip to content

feat(bin): surface validation daemon liveness at session start (land of upstream #1611) - #50

Open
sbracewell64 wants to merge 3 commits into
mainfrom
fm/land-daemon-liveness-1611
Open

feat(bin): surface validation daemon liveness at session start (land of upstream #1611)#50
sbracewell64 wants to merge 3 commits into
mainfrom
fm/land-daemon-liveness-1611

Conversation

@sbracewell64

Copy link
Copy Markdown
Owner

What this is

A landing of upstream contribution kunchenguid/firstmate#1611 ("feat(bin): surface validation daemon liveness at session start") onto this fork's trunk, so the running fleet actually gets it.
The change was already reviewed upstream; this branch does not redesign or re-review it.
Contribution kunchenguid#1611 remains open and untouched upstream, on the maintainer's schedule.

Why it matters

Nothing in the running code checks the shared validation daemon at all - every daemon reference in live bin/ is the away-mode daemon, a different thing.
On 2026-08-06 that daemon was found dead, last provably alive about 19 hours earlier, and nothing surfaced it; a worker stumbled into it mid-task and lost its turn.
Recorded history is three prior deaths that stayed down 8 hours, 2.5 days and 2.9 days, all undetected.

What it carries

The contribution's three commits, cherry-picked whole: bin/fm-validation-daemon-lib.sh, the hook in bin/fm-bootstrap.sh, plus AGENTS.md, docs/configuration.md, bootstrap-diagnostics guidance, and tests.
The library is carried in full, not trimmed to the bootstrap call, because a follow-up task consumes it as a shared component.

Three text conflicts were reconciled by hand, all of them adjacency between trunk work and the contribution, none of them ambiguous in intent:

  • AGENTS.md and .agents/skills/bootstrap-diagnostics/SKILL.md - the trunk's diagnostic-line list has since gained MODEL_REGISTRY, MODEL_PRICE, MODEL_VERIFY, ADMISSION_CONTROL and WAKE_LEDGER. Kept the trunk list and inserted VALIDATION_DAEMON at the contribution's chosen position.
  • tests/fm-bootstrap.test.sh - trunk added new cases at the same end of the file. Kept both sets and both runner entries.

Verified afterwards: the added and removed lines of this branch's diff against the trunk are identical to the contribution's own diff against its base, differing only in surrounding context.

Delivery disclosure

This shipped direct-PR without the no-mistakes pipeline, under the captain's time-boxed authorization of 2026-08-04.

  • It therefore carries no attestation marker, and none is implied anywhere in this branch.
  • It has had no automated code review and no pipeline gates.
  • The verification below is what was actually run, reported as-is, including every failure.

Verification

The check fires - negative control witnessed first

The daemon was not stopped. Another lane's runs share it, so the dead cases were exercised against a fixture daemon root, as the brief permits, while the healthy case used the real running daemon.

Healthy case, real daemon (pid 500169, alive):

REAL DAEMON pid 500169 IS ALIVE
parsed pid: 500169
quiet-report output: []
FM_BOOTSTRAP_VERBOSE_FACTS=1 -> BOOTSTRAP_INFO: validation daemon alive (pid 500169, up 12m)

End to end through bin/fm-bootstrap.sh with NM_HOME at the real root: silent, as intended.

The recorded JSON pid-file trap is handled. The naive read fails as documented, and kunchenguid#1611's parser does not:

$ kill -0 "$(cat ~/.no-mistakes/daemon.pid)"
/bin/bash: kill: `{"pid":500169,"started_at":"2026-08-06T11:47:00Z"}': not a pid or valid job spec

Dead and degenerate cases, fixture root (the pid used was a genuinely reaped process, confirmed dead before use):

STALE PID, NO PROCESS -> VALIDATION_DAEMON: down - recorded pid 1237025 is not running; started 2.4d ago, last active 19.0h ago
NO PID FILE           -> VALIDATION_DAEMON: down - no pid file at <root>/daemon.pid; last active 19.0h ago
MALFORMED PID FILE    -> VALIDATION_DAEMON: unknown - cannot read a pid from <root>/daemon.pid; last active 19.0h ago
ZERO PID              -> VALIDATION_DAEMON: unknown - cannot read a pid from <root>/daemon.pid; last active 19.0h ago
NONEXISTENT ROOT      -> (silent)

A stale pid record with no live process reports down, distinctly from healthy and distinctly from unknown; it is never read as healthy.
End to end through bin/fm-bootstrap.sh with NM_HOME at the dead fixture:

VALIDATION_DAEMON: down - recorded pid 1237025 is not running; started 2.4d ago, last active 19.0h ago

Suites and gates

Check Result
bin/fm-lint.sh pass (ShellCheck 0.11.0, pinned 0.11.0)
bin/fm-doc-audience-check.sh pass (surfaces=72 local_links=213)
bin/fm-test-run.sh --check-coverage pass (total=133 parallel=24 serial=98 serial_shards=4 herdr=11)
tests/fm-bootstrap.test.sh pass, 31 ok, 0 not ok
bin/fm-test-run.sh --lane portable-parallel-1 pass
bin/fm-test-run.sh --lane portable-parallel-2 pass
--lane portable-serial-1of4 366 ok, 1 not ok
--lane portable-serial-2of4 402 ok, 4 not ok
--lane portable-serial-3of4 370 ok, 4 not ok
--lane portable-serial-4of4 442 ok, 2 not ok

Every failure, and whether it pre-exists on the base - proven

All 11 serial-lane failures were re-run individually against a clean tree built from the base commit ed376cf (which does not contain bin/fm-validation-daemon-lib.sh at all). Ten reproduce there identically:

Failing case Suite On base ed376cf
Pi extension must surface an external healthy watcher as an owned-wake failure fm-pi-watch-extension fails identically
first inheritance transaction never reached its blocked write fm-remote-secondmate-lifecycle-e2e fails identically
a running harness-named foreground process must classify alive fm-tmux-agent-liveness fails identically
turn_end drive failed: node:internal/modules/esm/get_format:219 fm-busy-adapter-wiring fails identically
Pi guard must inject once for no-tool and multi-tool logical runs fm-turnend-guard fails identically
restart did not attach to the verified healthy peer fm-watcher-lock fails identically
MISSING diagnostic did not appear at all fm-session-start fails identically
launch must come from launch_template, ghost-text suppression included fm-launch fails identically
Pi calm home resolution failed: node:internal/modules/esm/get_format:219 fm-calm-pi-extension fails identically
the tmux task shell did not become ready fm-backend-tmux-smoke fails identically

The eleventh needed more work and is not caused by this change either:

  • fm-secondmate-harness - "Claude secondmate with a fresh beacon should use auto-arm supervision". This passed on the base tree 3/3 and failed on this branch 2/2, so it was investigated rather than waved through. The captured output is the worktree-tangle guard: WORKTREE TANGLE - PRIMARY CHECKOUT IS ON A FEATURE BRANCH ... is on 'fm/land-daemon-liveness-1611', not its default branch 'main'. The assertion requires empty output, and the guard fires purely on the checkout's branch name. Proven by checking the base tree out onto a branch of the same name and re-running: it then fails identically with none of this change's code present. It is a property of running that suite from any feature branch, not a regression here.

One disclosed caveat about my own test run

On a first pass I ran the four serial shards concurrently on one machine, which the runner explicitly warns against. That produced three extra failures of the form VALIDATION_DAEMON: down - no pid file at /tmp/fm-test-absent-nm-home/daemon.pid. The cause is real and worth recording: tests/lib.sh in this contribution pins NM_HOME at a path expected not to exist, but a suite elsewhere in the repo invokes the real no-mistakes CLI, which materializes that root; once it exists, suites asserting exact bootstrap silence see a VALIDATION_DAEMON line. Re-running the shards sequentially, cleaning that path between shards, produced zero such failures, which is why they are not listed above. CI runs each shard on its own runner, so this does not arise there - but the sentinel-path approach to hermeticity is only as durable as nothing else creating that directory. Reported, not patched, since it belongs to the reviewed contribution.

Known gap, inherent to the contribution and not to this landing

bin/fm-test-run.sh --changed refuses with no changed-test mapping for source path: bin/fm-validation-daemon-lib.sh. The mapping resolves a bin/ script by scanning tests/ for its basename, and the new tests exercise the library through bin/fm-bootstrap.sh without ever naming it. This is not a conflict-resolution artifact: upstream #1611's own head has zero references to that basename in tests/ and would refuse the same way. It does not affect CI, which runs --check-coverage and the named lanes rather than --changed, and --check-coverage passes. Reported rather than silently patched into this landing.

… start

Every shipping task depends on one shared no-mistakes validation daemon, and
nothing ever asked whether it was running. It died silently three times and
stayed down 2.5 days, 2.9 days and 8 hours before anything restarted it; nobody
noticed on any occasion, because a silent outage is indistinguishable from a
quiet fleet.

Bootstrap now reads the daemon's pid file and sends signal 0 once per run, and
reports the result alongside the existing diagnostics: a VALIDATION_DAEMON line
when the daemon is down or unmeasurable, silence when it is healthy, and its
uptime as a BOOTSTRAP_INFO fact only under FM_BOOTSTRAP_VERBOSE_FACTS=1. A down
report carries how long the daemon has been down, taken from the last moment it
demonstrably wrote anything, so a 2.9-day outage reads as a 2.9-day outage.

The check observes only. It never starts, stops, restarts or reconfigures the
daemon, and it never invokes the no-mistakes CLI, whose ordinary commands
auto-start a daemon as a side effect. Starting a down daemon stays firstmate's
decision.

Three outcomes stay distinct rather than collapsing into two. The pid file is
JSON, not a bare integer, so piping it into a signal check reports a live daemon
as down; an unreadable pid file is therefore reported as unknown, because
calling it down manufactures false alarms and calling it alive recreates the
silence this check exists to end. A recorded pid of 0 is unknown for the same
reason: kill -0 0 signals the caller's own process group.

Tests cover alive, down, missing pid file, and four malformed pid files, and
construct the dead pid positively rather than inferring down from a live daemon.
tests/lib.sh pins NM_HOME at a nonexistent root so the machine's real daemon
cannot leak into any suite that asserts exact bootstrap output.

(cherry picked from commit ad100b8)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant