fix(diagnostics): categorise every finding so a red run says which of four things it is (#3312)

[Xore]

Fixes #3312.

diagnostics.yml failed 12 scheduled runs in a row and nobody triaged it, which means it carried no signal at all. The findings were not all wrong — several were correct — but four unrelated things all produced the same output: a red X and an ::error:: line that named none of them. A run that lists five unrelated findings under one heading gets triaged by ignoring it.

This PR decides each reported finding, makes the check able to tell the categories apart, and keeps every one of them fatal except a declared, deliberate absence.

1. Classification of every reported failure

Reported	Category	Why	What this PR does
metrics unavailable: no DASHBOARD_SERVICE_TOKEN in /opt/stacks/honeypot-dashboard/.env	runner-config gap	The token was never missing. It lives in a root:root 0600 .env and the job ran as github-deploy-runner, so the job's own sed read failed on permission and the failure was reported as an absent key — a claim about the host it had no way to make. #3338's root-owned helper fixes the read, but the lane stayed red until an operator applied the grant, which a workflow cannot do for them.	Filed as runner-config, still fatal, naming the one command that fixes it.
OIDC discovery … returned 403, not 200	runner vantage	Cloudflare answers 403 to GitHub-hosted runner address ranges while serving the same URL 200 to the VPS, the homeserver and a residential client (verified 2026-09-26).	Already fixed on main (#3338: probe from the VPS). Now unmeasured (no HTTP response at all) vs fault (answered, and not 200) are separate findings.
libvirt socket not found at /var/run/libvirt/libvirt-sock	real host regression	The 2026-09-23 reboot let the modular per-driver virt* units win the Conflicts= race against libvirtd and take the socket with them. #3335's hypothesis — that #3135's docker pause caused it — is disproved: that pauses containers, never libvirt.	Stays a FAIL, exit 1.
libvirt network 'sandbox' does not exist / 'honeypot-sandbox' does not exist / 'honeypot-sandbox-strict' nwfilter is missing	check bug, compounded by a possible stand-down	Bare virsh as a non-root caller resolves to qemu:///session — an empty per-user instance — so active objects on qemu:///system read as missing. Fixed on main (#3338 pins the URI). Separately, these are absence invariants, and absence has two causes: a real regression, or a deliberate stand-down to free RAM/CPU for a training leg, which is standing practice here (#3135). Before this PR both printed the same FAIL.	URI pin kept; absence now carries a dated declaration (below), so a stand-down is EXPECT and exits 0 while a regression stays FAIL and exits 1.
hp-unsloth-studio has no cap_drop: ALL and is on neither list	real gap, already fixed on main	It is a long-lived studio container that needs to read and write a xore-owned 0600 notebook as that user, so cap_drop: ALL alone broke it. #3339 gives it cap_drop: ALL plus cap_add: [DAC_OVERRIDE] (and no-new-privileges), and #3341 dropped the tracked-gap list entry. Live: CapDrop=[ALL], CapAdd=[CAP_DAC_OVERRIDE].	No exceptions-list entry added — the container spec was fixed instead, per the brief.

The exceptions list

No new entry was added to any exceptions or tracked-gaps list, so there is nothing to justify by reference. The only audit lists touched are unchanged in membership.

The mechanism I did add is a stand-down declaration, which is the same idea with a harder deadline: scripts/sandbox-standdown.sh declare --issue '#NNNN' --until YYYY-MM-DD --reason '…', written to /etc/apiary/sandbox-standdown (root-owned 0644, because the audit runs unprivileged). An exception with an owner issue and an expiry is the reference — every declaration must cite one, and issue: 1 without the # is rejected by both the writer and the reader. Expired, malformed, unparseable-date, and more-than-14-days-out declarations are all FAIL themselves, so the file cannot rot into a permanent excuse. A declaration excuses absence only: a libvirt network that exists and forwards, or a planted FORWARD ACCEPT for virbr-sandbox, is still a fault with one in force (both are pinned by tests).

2. The vps job, broken down

Nobody had done this. From run 36219375542 (2026-09-23) and the runs since:

• Configure deployment key — passes.
• Inspect the VPS edge — the sentinel printed, so SSH delivered the whole report. Passed. Compose ps, eve file sizes and timestamps, the ruleset age check, disk and WireGuard peers were all read successfully.
• Traefik origin certificate expiry (#3328) — passed; the certificate was readable and not near expiry.
• OIDC discovery is reachable (#1225) — the only failure: OIDC discovery HTTP status: 403. That 403 is Cloudflare's answer to the GitHub-hosted runner's address range, not OIDC's health (see the table above). The VPS job has been success in the last four scheduled runs, since fix(ops): make Diagnostics truthful again — runner vantage, virsh URI, token access, libvirt stack #3338 moved the probe to the VPS.

So the vps job's red X was one runner-vantage finding wearing the same uniform as the home job's. It is now filed as such.

3. What changed

Every finding is categorised, in the annotation's own title and as one row per finding in a ledger each job prints last:

category	meaning	fatal on a scheduled run
fault	a real regression	yes
runner-config	the check could not run because of how the runner or this repo's environment is configured	yes
unmeasured	the check did not run, and that is not a pass	yes
expected	a declared, deliberate absence	no

runner-config and unmeasured staying fatal is deliberate and is the same position scripts/verify-deploy.sh already takes with its exit 2. Folding "could not tell" into a pass is what makes a check worse than not having it — #3283 is what that costs when it is wrong (Elasticsearch at 1000/1000 shards, every sensor dead-lettered for six days, in a lane whose only question is whether the pipeline is flowing). What changes is that it is tellable: ::error title=runner-config: … names the fix, and the ledger's counts make "0 faults, 1 runner-config gap" a healthy host with one misconfigured lane rather than an incident.

The vocabulary lives in scripts/diagnostics-lib.sh (new), which every step sources. Five inline copies in YAML would be five things to keep in step; a test asserts no alert/note call site is uncategorised and that no call site uses a category the library does not define.

Two check bugs fixed while making the categories honest:

• ss -tlnp | grep ':22 ' exits 1 both when nothing listens on :22 and when ss could not run at all, so the sshd check had never once answered on this host — it reported "could not confirm" about a host whose answer was simply "nothing is listening". It now distinguishes OK from UNMEAS, and UNMEAS is fatal for the isolation barriers. "The FORWARD chain could not be read" and "the FORWARD chain is not DROP" are different findings, and only the second says the host is unprotected.
• docker ps | grep -E '^(hp-|sbx-)' exits 1 on a host with no stack containers, reported as could not enumerate containers (docker ps failed) — a claim about the tool printed when the truth was a claim about the deployment. Both are faults; only one is fixable by reinstalling docker.

The audit runs from the job's checkout (#2908), not from /opt/stacks/apiary. That directory is refreshed only by deploy.yml, which is workflow_dispatch-only, so the audit that kept naming things #3338 had already fixed was a copy that had not been redeployed. A check that audits the host with a stale copy of its own question cannot report a fix as fixed. The deployed tree is still diffed against origin/main and reported when it drifts — drift is still a real finding for everything else on the host that runs a deployed script.

One wrong-signal bug fixed in passing: the cert step's < 30 days branch was report "::error::…" followed by [ "${{ github.event_name }}" = "schedule" ] && exit 1. That compound was the last command in the block, so a manual run exited 1 anyway — a workflow_dispatch run went red for a warning it was explicitly meant to report only. alert() has the schedule gate inside it, so the intent is now the behaviour.

4. Verification

tests/docs/test_3312_signal_categories.py (new, 35 tests) drives the real isolation-audit.sh and the real scripts/diagnostics-lib.sh and the real run: body of the isolation step against synthesised hosts (every external command is a PATH stub, unshare -rm + a bind mount for /var/run/libvirt). It is behaviour, not string-matching: a finding is asserted by its category column, not by a substring.

Before/after exit codes, same host, only the declaration differs (origin/main:scripts/isolation-audit.sh vs this branch):

case	before	after
healthy host, no declaration	0	0
host broken (libvirt + both networks + nwfilter gone), no declaration	1	1
same host, live declaration (#3135, until 2026-10-04)	1	0
same host, expired declaration (until 2020-01-01)	1	1

The expected-state case is the only one that changes, and it changes because the host now has something to say. A real fault is still 1; a rot exception is still 1.

The expected-state run says so in its own output, rather than going quiet:

== Declared stand-down of the sandbox isolation stack ==
  EXPECT  sandbox isolation stack is deliberately absent (declared stand-down, #3135, until 2026-10-04: …). Absent sandbox objects below are EXPECTED, not faults; anything else that breaks below is still a FAIL, and the declaration does not cover it
  EXPECT  libvirt network 'sandbox' does not exist (expected active, isolated) -- EXPECTED, not a fault (declared stand-down, #3135, until 2026-10-04: …)
isolation-audit: categories -- 0 unmeasured, 6 expected-by-declaration, 0 triaged gap(s), 0 fault(s)
isolation-audit: VERDICT PASS -- every check that could be measured agrees with the invariants (6 object(s) excused by a live declaration)

Live run on this host (a scratch box, not the production runner), showing the categories on real findings — note that no hp-* or sbx-* container exists on this host at all is now a claim about the deployment, distinct from a docker failure:

== Declared stand-down of the sandbox isolation stack ==
  --      no stand-down declaration at /etc/apiary/sandbox-standdown -- the full sandbox stack is expected here
== Sandbox libvirt networks: no <forward> ==
  FAIL    libvirt network 'sandbox' does not exist (expected active, isolated) -- if the sandbox stack is meant to be up, …
== Phase 0 iptables barrier (guarded sandbox bridges) ==
  OK      FORWARD default policy is DROP
  OK      nothing explicitly ACCEPTs virbr-sandbox traffic
  OK      nothing explicitly ACCEPTs virbr-hpsbx traffic
== Host posture (reports only, does not fix) ==
  OK      sshd is not listening on a honeypot-facing address
  FAIL    libvirt socket not found at /var/run/libvirt/libvirt-sock -- … (#3338)
isolation-audit: categories -- 0 unmeasured, 0 expected-by-declaration, 0 triaged gap(s), 6 fault(s)
isolation-audit: VERDICT FAIL -- 6 fault(s) above: …

The isolation step, executed as the runner executes it (its literal run: body, with a stub audit):

audit says	step files	step exit
0 unmeasured … 1 fault(s), exit 1	fault	1
2 unmeasured … 1 fault(s), exit 1	unmeasured	1
0 unmeasured … 0 fault(s), exit 0	ok	0

…each of which then renders in the job's ledger:

| category | check | finding |
| --- | --- | --- |
| fault | isolation audit (scripts/isolation-audit.sh) | isolation-audit: categories -- 0 unmeasured, 0 expected-by-declaration, 0 triaged gap(s), 1 fault(s) |

counts: 0 check(s) measured and in agreement, 1 fault(s), 0 runner-config gap(s), 0 unmeasured, 0 expected absence(s).

Gates: pytest tests/docs/ 556 passed, 1 xfailed, 17 subtests (521 before this branch; +35 new). actionlint -S warning clean. shellcheck -S warning clean on all three scripts, bash -n clean. check-doc-paths-exist.py, check-docs-reachable.py, check-doc-stale-paths.py, check-compose-env-docs.py, check-public-leaks.py all pass.

5. Constraints

• No check was deleted. Every finding in the old workflow still exists and still fires; the ones that used to be report-only are still report-only.
• No blanket || true or continue-on-error anywhere.
• No check was changed to ask a different question. The exception is the expectation of the sandbox stack, which is what a stand-down legitimately changes, and it is now declared on the host with an owner issue and an expiry rather than assumed. Two report-only branches were deliberately left report-only rather than promoted, because promoting them would have been changing what the check asks.
• Not merged — left open for review.

6. Two things this PR does not do

• The job stays red until an operator runs one command. sudo scripts/github-ci-runner/install-deploy-runner.sh --helpers-only applies the source-health helper and its grant without stopping the runner service. A merged PR cannot put host state on the box, so the metrics lane is still unmeasurable until that runs. It is now a runner-config finding that names the command, rather than a red X that reads as a broken ingest pipeline. I would rather leave it red than make "could not tell" mean "fine".
• No new CI check for cap_drop. I looked at adding a static compose-vs-exceptions check (7 services are not in hp-* scope or are on the audit's lists). It is a separate concern from diagnostics signal and a new gate in quality.yml is a larger surface than this issue asks for. Worth its own issue.

[github-actions]

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

OpenSSF Scorecard

Package	Version	Score	Details
actions/actions/checkout	3d3c42e5aac5ba805825da76410c181273ba90b1	🟢 6.6	Details Check Score Reason Maintained 🟢 7 9 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 7 Code-Review 🟢 10 all changesets reviewed Dangerous-Workflow 🟢 10 no dangerous workflow patterns detected Binary-Artifacts 🟢 10 no binaries found in the repo Token-Permissions ⚠️ 0 detected GitHub workflow tokens with excessive permissions CII-Best-Practices ⚠️ 0 no effort to earn an OpenSSF best practices badge detected Pinned-Dependencies 🟢 3 dependency not pinned by hash detected -- score normalized to 3 Fuzzing ⚠️ 0 project is not fuzzed License 🟢 10 license file detected Packaging ⚠️ -1 packaging workflow not detected Signed-Releases ⚠️ -1 no releases found Security-Policy 🟢 9 security policy file detected SAST 🟢 10 SAST tool is run on all commits Branch-Protection 🟢 5 branch protection is not maximal on development and all release branches

Scanned Files

• .github/workflows/diagnostics.yml