fix(fm-watch): read certsync health off host files, drop docker exec

[Freudator86]

Why

Since the honesty fix (#73), the certsync health check honestly reports that it
cannot run rather than silently reading as healthy. Reproduced live on
2026-08-05: docker compose exec against projects/hlr-certsync returns
permission denied on unix:///var/run/docker.sock, because this account is
deliberately not in the docker group. Net effect today: no read at all on
whether the certificate sync service is well.

Adding the account to the docker group, or fronting the socket with an
exec-capable proxy, was explicitly out of scope — exec through a proxy is
root-equivalent on the host and is not a scoped read however narrowly the proxy
is named.

What this does

certsync status already computes healthy/reason purely from the sqlite
state DB and the heartbeat JSON, plus a caller-supplied --daemon-state
argument (which this check has always passed as the literal running). There is
no container-only input to that computation. So instead of exec-ing into the
container, certsync_health_reason now runs certsync's own build_status
directly against those two files on the host, via python3 with
PYTHONPATH=$FM_CERTSYNC_SRC (default $FM_CERTSYNC_PROJECT/src).

This removes the need for docker access entirely — no socket, no exec, no
docker-group membership, no proxy — and reproduces the exact healthy/reason
JSON the exec path produced (verified byte-for-byte against real certsync state
files).

certsync exposes the two files on a readable host path in a paired PR in the
certsync repo (heavyliftrental/hlr-certsync, branch
fm/hlr-certsync-health-read-access): it backs /var/lib/hlr-certsync with a
host bind mount instead of a docker-managed named volume. Deploy order: that
compose PR lands and is deployed first (see its docs/deploy.md → "State host
path"); this check reads the exposed files.

Liveness: heartbeat freshness gate (engineering decision — please confirm)

Reading frozen files loses the one liveness signal exec gave for free: exec
failed when the container was down. Without replacing it, a stopped container
whose last-written files still say success would read as healthy off stale
files — the exact silent-healthy failure #73 exists to prevent.

The heartbeat timestamp is in the exposed files, so I reinstated liveness as a
freshness bound: FM_CERTSYNC_HEARTBEAT_MAX_AGE (default 7200s = 2× the
3600s worst-case sync interval; 0 disables). A healthy:true reading whose
heartbeat has gone stale now reads as unhealthy: heartbeat stale (...), never
quiet. The daemon rewrites the heartbeat on every successful sync pass
(including no-op passes), so a live service never trips the bound.

Two consequences worth a reviewer's eye:

• This also closes a pre-existing blind spot: under the exec path, a run of
  failing syncs (heartbeat frozen at the last success, last_run_state always
  success, no poison) read as healthy. It now surfaces once stale.
• The 7200s default is a judgment call. Set FM_CERTSYNC_HEARTBEAT_MAX_AGE=0
  to restore the raw build_status verdict with no freshness gate.

Operator step (not done here — deployment is out of scope)

After both PRs merge, on hlr-1, provision the readable state dir and
recreate the container so the bind mount takes effect:

sudo mkdir -p /var/lib/hlr-certsync
sudo chown 10001:10001 /var/lib/hlr-certsync
sudo chmod 0755 /var/lib/hlr-certsync
# migrate existing state out of the old named volume if present, then:
cd ~/projects/hlr-certsync && docker compose up -d --force-recreate certsync

(Full migration notes in the certsync PR's docs/deploy.md.)

Tests

tests/fm-watch-triage.test.sh certsync cases rewritten to the file-read model
(a hermetic fake hlr_certsync package stands in for the source tree), with new
coverage for the freshness gate (healthy + stale heartbeat → unhealthy) and
the source-unavailable preflight. All 7 certsync cases plus the run-bounded
regression pass locally.

Merge note

This repo forbids squash merges — merge with a merge commit or rebase, not
squash.

[Freudator86]

Superseded by #78, raised through the sanctioned no-mistakes channel.