ci(frontend-next): measured coverage floor + test-discovery guard (#3318)

[Xore]

Closes #3318.

frontend-next ran vitest and Playwright in quality.yml and collected no coverage, so there was no baseline. A change could delete tested behaviour and CI stayed green — the only thing that noticed was a reviewer noticing the test that asserted it was gone.

The measured numbers

Taken on the node:22-alpine image this job actually runs (node v22.23.2, npm 10.9.8, vitest 4.1.11), 24/24 files and 215/215 cases green:

metric	covered	total	pct
lines	806	7359	10.95%
branches	346	7250	4.77%
statements	859	8452	10.16% (recorded, not gated)
functions	107	2666	4.01% (recorded, not gated)

127 files of src/. That is low, and it is the real number. The suite concentrates on src/lib server logic; most of src/routes and src/components is held by the Playwright matrix, which the v8 provider cannot see. No threshold was imported and no number was picked to look respectable — a 60/60/60/60 baseline would have been red on arrival and would have taught the gate nothing. coverage-baseline.json records what the suite measures, and tolerance is set to hold that.

The ratchet

Two independent gates, both from the committed baseline and this run's own measurement:

• tolerance.coveredCount = 0 — no covered line or branch may stop being covered. This is the half that catches deleted tested behaviour, and no shrinking-denominator trick can satisfy it (delete an untested file, the percentage goes up).
• tolerance.pctPoints = 1.0 — the overall percentage may not fall more than that. This catches a change that adds a lot of untested source.

At an ~11% baseline the second gate is the coarse one: it needs several hundred new uncovered lines to move, because covered/(pct − tol) is what the denominator has to exceed. That asymmetry is documented in the baseline, the README and the ratchet, and it is why the covered-count gate carries the weight.

Updating the floor is npm run coverage:baseline — a visible diff, never an autoUpdate, so a check cannot rewrite itself. scripts/tests/test_3318_coverage_ratchet.py pins coveredCount == 0 and bounds pctPoints in both directions, so a later PR cannot widen the tolerance without going red.

Proof the ratchet bites

One real test file (src/lib/credentialState.test.ts) temporarily moved out of the include globs, re-measured, then restored (git diff HEAD on it is empty — nothing in src/ is touched by this PR):

coverage-ratchet: 127 files under src/ (baseline 127)
  FAIL lines     10.64% (783/7359)  baseline 10.9526% (806/7359)  drop 0.3126pp / 23 covered
  FAIL branches  4.2621% (309/7250)  baseline 4.7724% (346/7250)  drop 0.5103pp / 37 covered
::error::coverage regression (#3318) ...
exit 1

Worth noting: 0.31pp of lines is inside the 1.0pp tolerance. The percentage gate alone would have passed that drop. Only the covered-count gate caught it — which is the evidence for why it exists, and why it is pinned at 0.

Proof the discovery guard bites

A canary src/lib/__discovery_canary.spec.ts — deliberately the .spec infix, which the vitest include globs do not name, so a guard that only knew .test would have passed it:

test-discovery: vitest collects 24 file(s)   ← unchanged; the canary is not in it
test-discovery: 26 test-shaped .ts/.tsx file(s) under ./
::error::1 test file(s) exist that no configured runner collects (#3318):
  - src/lib/__discovery_canary.spec.ts
exit 1

Canary removed, guard green again (25 files). Both proofs are now pinned as cases in scripts/tests/test_3318_coverage_ratchet.py (17 tests, auto-discovered by the existing scripts/tests lane — no workflow edit).

The guard asks the real runners (vitest list --filesOnly, playwright test --list) rather than reimplementing their globs. Both, because two runners own .ts/.spec.ts here — asking only vitest would flag e2e/dashboard.spec.ts, and the fix would be a suppression list. One trap needed handling: Playwright's collection imports every file its testMatch reaches, and importing e2e/fake-backend.test.mjs (a node:test module) writes a TAP banner to stdout ahead of the JSON — verified in the image this gate runs on. The report is read from a file via PLAYWRIGHT_JSON_OUTPUT_NAME.

Wiring

• @vitest/coverage-v8@^4.1.11 (matches the vitest the lockfile resolves) and test:coverage, test:discovery, coverage:ratchet, coverage:baseline scripts. Lockfile change is purely additive (171 insertions, 0 deletions) and re-resolved with npm 10 — the image's npm — so the existing "lockfile installs under the image's npm" check still passes.
• Coverage include is src/** only. Excluded: the tests, *.d.ts, and the generated routeTree.gen.ts. Verified: 127 files in the report = all 153 src/ .ts/.tsx minus 24 tests, novnc.d.ts and routeTree.gen.ts.
• npm test is untouched and stays uninstrumented — the price is a second run, and it is 6.9s against 6.9s (the cost is transform/import, not instrumentation) in a 45-minute budget.
• New steps in the existing frontend-next job. No new job. The artifact is frontend-next-coverage-${{ github.run_id }}-${{ github.run_attempt }} — the per-(run, attempt) naming from ci: name every artifact per run instead of overwriting a shared name (#3314) #3400, no overwrite, 7-day retention. Mirrored into its GitHub-hosted twin, because the twins are byte-identical by convention and a ratchet that only ran on the self-hosted executor would skip every degraded day.
• coverage/ is gitignored; the committed record is coverage-baseline.json.
• No new action — the upload reuses the SHA already pinned in this file.

Checks

• npm run typecheck, npm test (215/215), npm run build, git diff --exit-code src/routeTree.gen.ts — all clean on the image's node 22.
• zizmor: identical finding set to pristine HEAD — the single already-allowlisted dangerous-triggers advisory. No new finding, nothing new allowlisted. actionlint clean; the workflow's own unquoted-name-with-# guard clean.
• scripts/tests full suite: 296 tests, 2 failures — both in test_compose_drift_watch_sweep.py, both reproduced identically on a pristine HEAD checkout (this box cannot create the unreadable-path precondition those two need). Unrelated to this PR. My 17 pass.

Left alone deliberately

e2e/fake-backend.test.mjs is a node:test file that no configured runner collects either — a pre-existing instance of exactly what this guard looks for. Wiring it up is a different runner in a different tier, so the guard is scoped .ts/.tsx rather than being a new gate that is red on day one. Flagging it here rather than fixing it inside this PR.

Not merged

Do not merge — needs review.

[github-actions]

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

OpenSSF Scorecard

Scorecard details

Package	Version	Score	Details
npm/@bcoe/v8-coverage	1.0.2	Unknown	Unknown
npm/@vitest/coverage-v8	5.0.2	Unknown	Unknown
npm/@vitest/istanbul-lib-coverage	1.0.2	Unknown	Unknown
npm/@vitest/istanbul-lib-report	1.0.2	Unknown	Unknown
npm/@vitest/mocker	5.0.2	Unknown	Unknown
npm/@vitest/spy	5.0.2	Unknown	Unknown
npm/ast-v8-to-istanbul	1.0.7	Unknown	Unknown
npm/magicast	0.5.5	Unknown	Unknown
npm/tinybench	6.2.0	Unknown	Unknown
npm/tinyrainbow	3.1.1	Unknown	Unknown
npm/vitest	5.0.2	Unknown	Unknown
npm/why-is-node-running	3.2.2	🟢 3.8	Details Check Score Reason Dangerous-Workflow 🟢 10 no dangerous workflow patterns detected Code-Review ⚠️ 1 Found 4/30 approved changesets -- score normalized to 1 Pinned-Dependencies ⚠️ 0 dependency not pinned by hash detected -- score normalized to 0 Binary-Artifacts 🟢 10 no binaries found in the repo Token-Permissions 🟢 9 detected GitHub workflow tokens with excessive permissions Maintained ⚠️ 0 0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0 CII-Best-Practices ⚠️ 0 no effort to earn an OpenSSF best practices badge detected Fuzzing ⚠️ 0 project is not fuzzed License 🟢 10 license file detected Packaging ⚠️ -1 packaging workflow not detected Signed-Releases ⚠️ -1 no releases found Branch-Protection ⚠️ 0 branch protection not enabled on development/release branches Security-Policy ⚠️ 0 security policy file not detected SAST ⚠️ 0 SAST tool is not run on all commits -- score normalized to 0

Scanned Files

• arcane/home/honeypot-dashboard/frontend-next/package-lock.json