feat(bin): surface validation daemon liveness at session start (land of upstream #1611) - #50
Open
sbracewell64 wants to merge 3 commits into
Open
feat(bin): surface validation daemon liveness at session start (land of upstream #1611)#50sbracewell64 wants to merge 3 commits into
sbracewell64 wants to merge 3 commits into
Conversation
… start Every shipping task depends on one shared no-mistakes validation daemon, and nothing ever asked whether it was running. It died silently three times and stayed down 2.5 days, 2.9 days and 8 hours before anything restarted it; nobody noticed on any occasion, because a silent outage is indistinguishable from a quiet fleet. Bootstrap now reads the daemon's pid file and sends signal 0 once per run, and reports the result alongside the existing diagnostics: a VALIDATION_DAEMON line when the daemon is down or unmeasurable, silence when it is healthy, and its uptime as a BOOTSTRAP_INFO fact only under FM_BOOTSTRAP_VERBOSE_FACTS=1. A down report carries how long the daemon has been down, taken from the last moment it demonstrably wrote anything, so a 2.9-day outage reads as a 2.9-day outage. The check observes only. It never starts, stops, restarts or reconfigures the daemon, and it never invokes the no-mistakes CLI, whose ordinary commands auto-start a daemon as a side effect. Starting a down daemon stays firstmate's decision. Three outcomes stay distinct rather than collapsing into two. The pid file is JSON, not a bare integer, so piping it into a signal check reports a live daemon as down; an unreadable pid file is therefore reported as unknown, because calling it down manufactures false alarms and calling it alive recreates the silence this check exists to end. A recorded pid of 0 is unknown for the same reason: kill -0 0 signals the caller's own process group. Tests cover alive, down, missing pid file, and four malformed pid files, and construct the dead pid positively rather than inferring down from a live daemon. tests/lib.sh pins NM_HOME at a nonexistent root so the machine's real daemon cannot leak into any suite that asserts exact bootstrap output. (cherry picked from commit ad100b8)
(cherry picked from commit 404838f)
(cherry picked from commit ab0ad0c)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
A landing of upstream contribution kunchenguid/firstmate#1611 ("feat(bin): surface validation daemon liveness at session start") onto this fork's trunk, so the running fleet actually gets it.
The change was already reviewed upstream; this branch does not redesign or re-review it.
Contribution kunchenguid#1611 remains open and untouched upstream, on the maintainer's schedule.
Why it matters
Nothing in the running code checks the shared validation daemon at all - every
daemonreference in livebin/is the away-mode daemon, a different thing.On 2026-08-06 that daemon was found dead, last provably alive about 19 hours earlier, and nothing surfaced it; a worker stumbled into it mid-task and lost its turn.
Recorded history is three prior deaths that stayed down 8 hours, 2.5 days and 2.9 days, all undetected.
What it carries
The contribution's three commits, cherry-picked whole:
bin/fm-validation-daemon-lib.sh, the hook inbin/fm-bootstrap.sh, plusAGENTS.md,docs/configuration.md,bootstrap-diagnosticsguidance, and tests.The library is carried in full, not trimmed to the bootstrap call, because a follow-up task consumes it as a shared component.
Three text conflicts were reconciled by hand, all of them adjacency between trunk work and the contribution, none of them ambiguous in intent:
AGENTS.mdand.agents/skills/bootstrap-diagnostics/SKILL.md- the trunk's diagnostic-line list has since gainedMODEL_REGISTRY,MODEL_PRICE,MODEL_VERIFY,ADMISSION_CONTROLandWAKE_LEDGER. Kept the trunk list and insertedVALIDATION_DAEMONat the contribution's chosen position.tests/fm-bootstrap.test.sh- trunk added new cases at the same end of the file. Kept both sets and both runner entries.Verified afterwards: the added and removed lines of this branch's diff against the trunk are identical to the contribution's own diff against its base, differing only in surrounding context.
Delivery disclosure
This shipped direct-PR without the no-mistakes pipeline, under the captain's time-boxed authorization of 2026-08-04.
Verification
The check fires - negative control witnessed first
The daemon was not stopped. Another lane's runs share it, so the dead cases were exercised against a fixture daemon root, as the brief permits, while the healthy case used the real running daemon.
Healthy case, real daemon (pid 500169, alive):
End to end through
bin/fm-bootstrap.shwithNM_HOMEat the real root: silent, as intended.The recorded JSON pid-file trap is handled. The naive read fails as documented, and kunchenguid#1611's parser does not:
Dead and degenerate cases, fixture root (the pid used was a genuinely reaped process, confirmed dead before use):
A stale pid record with no live process reports
down, distinctly from healthy and distinctly fromunknown; it is never read as healthy.End to end through
bin/fm-bootstrap.shwithNM_HOMEat the dead fixture:Suites and gates
bin/fm-lint.shbin/fm-doc-audience-check.shsurfaces=72 local_links=213)bin/fm-test-run.sh --check-coveragetotal=133 parallel=24 serial=98 serial_shards=4 herdr=11)tests/fm-bootstrap.test.shbin/fm-test-run.sh --lane portable-parallel-1bin/fm-test-run.sh --lane portable-parallel-2--lane portable-serial-1of4--lane portable-serial-2of4--lane portable-serial-3of4--lane portable-serial-4of4Every failure, and whether it pre-exists on the base - proven
All 11 serial-lane failures were re-run individually against a clean tree built from the base commit
ed376cf(which does not containbin/fm-validation-daemon-lib.shat all). Ten reproduce there identically:ed376cffm-pi-watch-extensionfm-remote-secondmate-lifecycle-e2efm-tmux-agent-livenessnode:internal/modules/esm/get_format:219fm-busy-adapter-wiringfm-turnend-guardfm-watcher-lockfm-session-startfm-launchnode:internal/modules/esm/get_format:219fm-calm-pi-extensionfm-backend-tmux-smokeThe eleventh needed more work and is not caused by this change either:
fm-secondmate-harness- "Claude secondmate with a fresh beacon should use auto-arm supervision". This passed on the base tree 3/3 and failed on this branch 2/2, so it was investigated rather than waved through. The captured output is the worktree-tangle guard:WORKTREE TANGLE - PRIMARY CHECKOUT IS ON A FEATURE BRANCH ... is on 'fm/land-daemon-liveness-1611', not its default branch 'main'. The assertion requires empty output, and the guard fires purely on the checkout's branch name. Proven by checking the base tree out onto a branch of the same name and re-running: it then fails identically with none of this change's code present. It is a property of running that suite from any feature branch, not a regression here.One disclosed caveat about my own test run
On a first pass I ran the four serial shards concurrently on one machine, which the runner explicitly warns against. That produced three extra failures of the form
VALIDATION_DAEMON: down - no pid file at /tmp/fm-test-absent-nm-home/daemon.pid. The cause is real and worth recording:tests/lib.shin this contribution pinsNM_HOMEat a path expected not to exist, but a suite elsewhere in the repo invokes the realno-mistakesCLI, which materializes that root; once it exists, suites asserting exact bootstrap silence see aVALIDATION_DAEMONline. Re-running the shards sequentially, cleaning that path between shards, produced zero such failures, which is why they are not listed above. CI runs each shard on its own runner, so this does not arise there - but the sentinel-path approach to hermeticity is only as durable as nothing else creating that directory. Reported, not patched, since it belongs to the reviewed contribution.Known gap, inherent to the contribution and not to this landing
bin/fm-test-run.sh --changedrefuses withno changed-test mapping for source path: bin/fm-validation-daemon-lib.sh. The mapping resolves abin/script by scanningtests/for its basename, and the new tests exercise the library throughbin/fm-bootstrap.shwithout ever naming it. This is not a conflict-resolution artifact: upstream#1611's own head has zero references to that basename intests/and would refuse the same way. It does not affect CI, which runs--check-coverageand the named lanes rather than--changed, and--check-coveragepasses. Reported rather than silently patched into this landing.