ci: fail when a test or crate is reached by no runner (#194)

[vidaunited]

ci: fail when a test or crate is reached by no runner (#194)

npm test is npm run -ws --if-present test and cargo test is --workspace, so
two things are currently indistinguishable from a pass: a workspace with no test
script, and a crate outside the members list. Both runners then report green and
complete. draco.yml already carries a note from the last time this bit --
"An explicit list silently drifts: arm tests existed but CI never ran them" --
and this generalises that fix instead of repeating it per-workflow.

scripts/check-runner-coverage.mjs walks every test file, package.json and
Cargo.toml and asserts each is either reached by a runner or recorded as a
deliberate exclusion with a reason. Zero dependencies, static, <1s, no network.

It counts three invocation forms as "reached", all of which are in use here:

• a workspace matched by the root workspaces globs that has a test script
• a file named explicitly in a workflow, as vitest run <path> OR node <path>
• a workflow job with working-directory: <dir> plus its own npm test

The second and third were added after the first draft produced FALSE POSITIVES
on this repo: ci.yml's "SOTA integrity-attestation gate" runs its suites under
plain node, and pages.yml tests apps/web-ui via working-directory. Those 11
findings were wrong, and a checker that cries wolf gets deleted. The
working-directory match is deliberately coarse (workflow-level, not job-level):
a missed gap costs less than a false alarm that blocks CI on correct code.

The allowlist is a RATCHET, not a suppression list -- seeded with what is
already unreached today so this passes on main unchanged, and anything NEW
fails. Entries require a non-empty reason; an allowlist without reasons is a
mute button and the next reader cannot tell a decision from an oversight. A
stale entry also fails, so it cannot rot into blanket permission. Deleting
entries is the point.

Seeded state (measured on 1f3b870, all cross-checked against the workflows):
56 test files -- 45 root tests/ (npm run -ws excludes the root package),
9 services/apicompletions (outside packages/),
2 kimi-k3-harness (outside packages/, has its own
"test": "vitest run" that nothing invokes)
4 crates -- oo-agents 20, horizon 14, k3rs 9, k3-kernel-bench 1
(#[test] fns; each declares its own [workspace], so cargo is
behaving correctly -- nothing else runs them either)

I am not asserting these should run: kimi-k3-harness reads like a self-contained
experiment and nested [workspace] crates are a normal way to isolate a
dependency graph. The claim is only that nothing currently distinguishes
"deliberately not run" from "accidentally not run".

Proved able to fail, four ways, each verified to die for the right reason:
new unreached test file -> red, names the file and why
allowlist entry with no reason -> red
stale allowlist entry -> red
drop the node <path> form -> the 2 SOTA suites reappear as findings
(no syntax error; the fix is load-bearing)

actionlint clean. The added CI step uses no ${{ }} expressions.

[vidaunited]

@ruvnet — this one needs an Approve and run workflows click when you get a chance; all three runs (CI / Security / Real Tools) are sitting at action_required as a first-time-contributor gate, so nothing has executed yet.

Low-risk to approve, for what it's worth: the check is a static, zero-dependency Node script — no network, no install, <1s — and the allowlist is seeded with the current state so it passes on main unchanged. It can only fail on something newly unreached.

If you'd rather sanity-check it locally before spending CI minutes:

git fetch origin pull/195/head:pr-195 && git checkout pr-195
node scripts/check-runner-coverage.mjs   # exits 0 today

And to see it bite, add any test file nothing runs — e.g. __tests__/x.test.ts — and it exits 1 naming the file and why.

No rush, and happy to adjust the approach if you'd prefer this as a warning rather than a gate, or scoped to fewer directories to start.

[vidaunited]

@ruvnet — thanks for #193, that closes the scope gap cleanly.

I checked it against the original defect rather than assuming: the new pattern matches the real pre-fix root build:wasm string at b17c50b (wasm-pack build crates/kernel-wasm --target bundler --out-dir ../../packages/kernel-js/pkg --release) and rejects the three near-misses worth worrying about — the current fixed command, a correct --target nodejs build, and a legitimate bundler build that doesn't aim at kernel-js/pkg. It also runs rather than skipping alongside the runtime half: CI reports loader.test.ts (16 tests | 1 skipped), and the file carries 15 unconditional it( blocks plus the single skipIf, so the skip is the wasm-artifact guard and the new check is among the 15 that executed.

Nudging #195 one last time: it's still sitting at action_required and needs an Approve and run workflows click before anything runs. It's the runner-coverage ratchet from #194, independent of #193.

Genuinely fine to close it if it's not wanted — I'd rather that than have it linger, and I won't keep bumping either way.

[ruvnet]

Reviewed and independently verified. This is a well-built guard and the PR description is unusually honest about its own limits — the false-positive story (node <path> and working-directory forms added after the first draft produced 11 wrong findings) is exactly the right instinct: a checker that cries wolf gets deleted.

Verified all four failure modes myself, checking real exit codes (a pipe to tail masks them — I caught myself doing that mid-review):

scenario	exit
clean tree, merged with current main	0
new unreached test file (tools/orphan/__tests__/…)	1 — names the file and why
allowlist entry with an empty reason	1 — allowlist entry with an empty reason: tests["…"]
stale allowlist entry (path gone/now reachable)	1 — is now reachable (or gone) — drop it

It also still passes against main after everything merged today (@metaharness/avo, numeric-* in darwin, host-adapter changes): workspaces npm test visits: 42 · cargo workspace members: 5 · test files not reached: 56 (56 allowlisted) · crates not reached: 4. The ratchet held across a busy day without anyone touching the allowlist, which is the property that matters.

It corrected me. While reviewing #212 I claimed apps/web-ui's 71 tests "run on no CI runner." That was wrong — pages.yml runs them via working-directory: apps/web-ui + npm test. This checker classifies web-ui as reached and correctly keeps it out of the allowlist, i.e. it got right what I got wrong by hand, precisely because of the working-directory form you added after the false positives. I've posted a correction on #212.

The residual gap there is narrower than I first said and this PR doesn't claim to cover it: pages.yml triggers only on push to main, never pull_request, so web-ui tests give no pre-merge signal. "Reached by a runner" and "reached before merge" are different properties; this checker measures the first, which is what it says on the tin.

node scripts/check-runner-coverage.mjs, zero deps, <1s. Merging.