feat(ci): add a gate-liveness check — prove each quality gate actually measured something by wshallwshall · Pull Request #25 · MEFORORG/MessageFoundry

wshallwshall · 2026-07-28T19:13:04Z

Three defects across two of quality-advisory.yml's gates spent months green while measuring
nothing — or, in one case, measuring correctly and publishing a wrong number. Every one of them
looked like good news.

The rubric's anti-metric rule guards against trusting a number too much. Nothing guarded against
trusting a green check that never ran. This is that control.

Each measurement job now emits a JSON receipt as a job output; a new liveness job reads
toJSON(needs) and rules on all four. It is the only job there allowed to go red — no
continue-on-error, no || true. It still blocks nothing: these contexts are not required and must
never become required.

The distinction it rests on

Liveness is not "the gate found something." A clean repo legitimately has zero clones, and a check
that fires on good news gets muted — worse than not having it. Receipts count units examined
(files scanned, mutants processed, changed lines analysed), non-zero whenever the tool ran, whatever
it concluded. A gate with nothing to measure passes by saying so, with a reason.

Rule	Catches
Proof of execution (units > 0)	mutmut crashing before producing a mutant
"Nothing to measure" only when stated	the shallow-fetch empty report — visually identical to the honest case
Derived figures cross-checked against an independent source	`killed=0` — the gate ran fine, the number was wrong

Verified against real data, not fixtures

complexity 256 files scanned / 122 findings · clone 234 files / 39 clones · mutation
461 mutants — 87 killed, 19 survived, 355 not covered, parsed from the actual artifact of run
30308667584. All four failure modes replayed through the real CLI and caught with exit 1.

The adversarial review found this control carrying the same weakness it was built to catch

In three places. All fixed here, all regression-tested:

A dead coverage gate could pass as not-applicable. Reproduced: a valid-but-empty
coverage.xml makes diff-cover print the identical "No lines with coverage information in this
diff" as the legitimate case — its console template branches on whether any source has measured
lines and never consults the diff. Same string, opposite meaning. Now the receipt proves
coverage.xml measured files first, a missing file is failed, and verify() refuses a
not-applicable receipt from a job whose result is failure.
An empty mutmut results file reported a flawless score. LISTED=0 → killed=TOTAL,
survived=0. A 100% kill rate with nothing listed is a tool that produced no output. Now failed.
The reconciliation was algebraically blind to the count it protected. killed = total − listed,
so the sum reduces to survived + no_tests + other == listed — total cancels. Fixed by
cross-checking against mutmut's own progress counter: two independent derivations that must
agree (verified, derived 87 == reported 87). The sum's blindness now has its own test rather than
being quietly deleted.

Also here

Rubric §4.0 (the liveness rule) + a v0.11 entry recording the above honestly. Corrected two
miscounts of my own: three defects across two gates, not three, and killed=0 is a different
category. Pre-cutover slug rot cleared in the two files this touches that #21/#22 don't cover —
including HANDOFF-mutation-coverage.md, which instructed a future session to re-add the exact
repo-slug gate removed when mutation started running on PRs.

Verification

ruff check + format --check clean · zizmor and bandit real exit 0 · pytest 9128 passed, 797
skipped on the current tree (re-run after the markdown edits rather than assumed).

…y measured something Three defects across two of this workflow's gates spent months green. Two were gates measuring nothing: diff-coverage (a shallow fetch destroyed its merge base, and the empty report looked clean) and mutation (mutmut crashed before producing a mutant, and `|| true` made that green in 37s). The third was the close cousin — a gate that measured correctly and published a wrong number (`killed=0` from a grep for a line mutmut never prints). The rubric's anti-metric rule guards against trusting a NUMBER too much. Nothing guarded against trusting a GREEN CHECK THAT NEVER RAN. This is that control. Each measurement job now emits a JSON receipt as a job output; a new `liveness` job reads toJSON(needs) and rules on all four. It is the ONLY job here allowed to go red — no continue-on-error, no `|| true` — and it still blocks nothing, since these contexts are not required and must never become required. THE DISTINCTION IT RESTS ON: liveness is not "the gate found something". A clean repo legitimately has zero clones, and a check that fires on good news gets muted, which would leave us worse off than before. Receipts count units EXAMINED — files scanned, mutants processed, changed lines analysed — non-zero whenever the tool ran, whatever it concluded. A gate with nothing to measure passes by saying so explicitly, with a reason. VERIFIED AGAINST REAL DATA, not fixtures: complexity 256 files scanned / 122 findings, clone 234 files / 39 clones, mutation 461 mutants (87 killed, 19 survived, 355 not covered) parsed from the actual artifact of run 30308667584. All four failure modes replayed through the real CLI and caught. AN ADVERSARIAL REVIEW BEFORE MERGE FOUND THIS CONTROL CARRYING THE SAME WEAKNESS IT WAS BUILT TO CATCH, IN THREE PLACES. All fixed here: * A dead coverage gate could pass by claiming `not-applicable`. Reproduced: a valid-but-EMPTY coverage.xml makes diff-cover print the identical "No lines with coverage information in this diff" as the legitimate case, because its console template branches on whether any source has measured lines and never consults the diff. Same string, opposite meaning — the exact ambiguity this control exists to resolve, one layer up. Now the receipt proves coverage.xml measured files first, a missing file is `failed` rather than inapplicable, and verify() refuses a `not-applicable` receipt from a job whose result is `failure`. * An empty mutmut results file reported a FLAWLESS score. LISTED=0 gives killed=TOTAL, survived=0 — 461/461 killed, reconciling perfectly. A 100% kill rate with nothing listed is a tool that produced no output, not a triumph. Now `failed`. * The reconciliation was algebraically BLIND to the count it claimed to protect. Since killed = total - listed, the sum killed+survived+no_tests+other == total reduces to survived+no_tests+other == listed: total cancels, and any derived killed satisfies it. So the incident-3 test was replaying numbers the production path can no longer produce. Fixed by cross-checking killed against mutmut's OWN progress counter — two independent derivations that must agree (verified: derived 87 == reported 87). The sum's blindness is now asserted by its own test rather than quietly deleted, so nobody mistakes it for protection it does not give. Also here: `other` is counted independently rather than as a remainder (a remainder would make the sum true by construction), across mutmut's full status vocabulary so the check cannot redden a healthy run; a non-numeric killed count emits `failed` instead of malformed JSON; and the coverage receipt step gets BASE_REF, which it was missing. Slug rot carried along, since this commit is already in the file: the coverage job's "move it to the mirror if that cost bites" is gone — there is no mirror post-cutover. Adjacent to the sweep in #21, which does not touch this workflow.

…re-cutover rot in these files Documents the gate-liveness control and promotes it to a rule in the rubric, because the failure it addresses is general rather than specific to this repo: an advisory gate that silently stops measuring is worse than an absent one, since the scorecard still counts it. Code_Quality_Standards.md gains section 4.0 (the liveness rule) and a v0.11 history entry. The rule has three parts, each written from a real failure rather than from theory: prove execution via units EXAMINED, never units found (a clean repo reports zero and must still pass); "nothing to measure" is acceptable only when stated with a reason; and any derived headline figure must be cross-checked against an independently produced measurement of the same quantity. That third part was rewritten during review. It originally said only "the parts must sum to the whole, each counted independently" — true but insufficient, because a sum containing a derived term can be algebraically blind to that term, and ours was. The rubric now says what actually protects a number, and the v0.11 entry records that the control was found carrying the same weakness it was built to catch. Leaving that out would have been the more flattering and less useful choice. Corrected two miscounts of my own from v0.10 and docs/CI.md: it was three defects across TWO gates, not three gates, and the killed=0 bug is a different category (a gate that ran correctly and published a wrong number) rather than a third instance of measuring nothing. The distinction was already drawn correctly in the rules; only the narrative sentences conflated them. PRE-CUTOVER SLUG ROT, in the files this commit already touches and which the open sweep does not cover: * HANDOFF-mutation-coverage.md was worse than stale. It instructed a future session to gate the mutation job with `if: github.repository == 'MEFORORG/MessageFoundry'` "(free minutes)" — the exact repo-slug gate removed when mutation started running on PRs. Following it would have made the job a no-op on every PR while looking deliberate. Left an explicit do-not-re-add marker rather than deleting the sentence, so the reasoning survives. * Code_Quality_Standards.md had one live "mirror-nightly" description of how the mutation job runs. Corrected and annotated. DELIBERATELY NOT SWEPT in that file: the mirror references in the control-parity narrative and in the v0.8 history row are dated incident records — the account of the fixed PyPI-sdist leak, which is the rubric's own worked example. Rewriting them would falsify the record.

…turally unable to (#27) `freethread-smoke.yml` could not fail in any path. Every step was `continue-on-error`, every step after setup was gated on `steps.setup.outcome == 'success'`, AND the job itself was `continue-on-error`. So a runner that could not provision 3.14t skipped everything and reported success; a failing GIL assertion or smoke test was swallowed and reported success. Five runs, five successes, and nothing in that record could distinguish a canary that flew from one that never left the ground. It IS currently flying — run 30256316219 shows `sys._is_gil_enabled() = False` on a free-threading build with 73 tests passing. The defect was latent, not active, which is exactly why it needed fixing: nothing would have told us when it stopped. An early-warning tripwire that cannot warn is worse than no tripwire, because its silence is mistaken for good news. The header also claimed "a red canary is an informational red check". No red canary was reachable. WHAT CHANGED. The job-level `continue-on-error` is gone. It bought nothing — this workflow runs only on a weekly cron and manual dispatch, never on `pull_request`, so it produces no PR check and cannot gate a merge regardless — and it cost the entire signal. The steps stay `continue-on-error` so they all run and every outcome stays collectable; a terminal verdict step then rules on those outcomes and IS allowed to fail. Green now means the canary flew. Three verdicts, each executed locally against the extracted shell rather than reasoned about: * did-not-fly — 3.14t could not be provisioned. A dead tripwire, not a clean run. * regressed — install, the GIL assertion, or the smoke tests failed under 3.14t. * vacuous — pytest exited 0 having collected nothing. Same shape as the gates fixed in #25: success that measured nothing. All six scenarios verified: healthy exits 0; setup-dead, GIL-re-enabled, smoke-failed, install-failed and vacuous-pass each exit 1 with the right diagnosis and a step summary naming it. The smoke step also stops hiding pytest behind a pipe (`| tee` yields tee's status, not pytest's) and now records how many tests actually ran, so the verdict can say what flew rather than merely that nothing errored. tests/test_freethread_smoke_liveness.py pins the two properties that make the signal real — the verdict step exists and can fail — and the one that makes failing safe: the workflow never runs on a pull request.

…uld not see it (#31) The liveness control shipped in #25 had this workflow's own failure mode inside it. The complexity receipt proved liveness with `ruff --show-files`, which counts the files ruff would enumerate — true whether or not the DELTA step ran. So a delta that bailed on an unresolvable merge base emitted a `::notice`, exited 0, and left a green `measured` receipt behind. On a PR the delta IS the point of that job, and it could go quiet while the gate reported success. The delta now records its own outcome to c901-delta.env, written LAST on the success path so its absence proves the step died, and written explicitly on the merge-base bail path so a skip is distinguishable from a crash. The receipt rules on it: PR + delta ran -> measured, evidence names the merge base and the base/head finding counts PR + marker missing -> failed, "did not complete" PR + delta bailed -> failed, carrying the recorded reason cron / dispatch -> measured, "the PR delta does not apply on <event>" That last row is deliberate. On a non-PR event the delta legitimately does not run and the whole-repo triage IS the measurement; failing there would fire on good news, and a check that fires on good news gets muted -- which would leave us worse off than before. TWO BUGS FOUND BY EXECUTING THE RECEIPT SHELL, not by reading it: * The reason was silently discarded. The receipt SOURCES the marker, and the reason was written unquoted, so bash parsed a value containing spaces as a command and left the variable unset. The failure reported "no reason recorded" -- throwing away the only useful diagnostic, on the exact path that exists to provide one. Now single-quoted, and asserted. * My own new test was vacuous. It asserted the marker appeared at least twice in the step, which the bail path satisfies on its own (it writes two lines), so deleting the SUCCESS-path marker passed. Caught by mutating the workflow and watching the assertion fail to go red. It now partitions the step at the c901_delta.py call and requires a marker on both sides. All four receipt paths were executed against the extracted shell; all four negative probes on the new assertions now go red.

…est plan (#161) * fix(docs): unbreak main — six new mirror-prose hits from the master test plan main has been red since 4ea1501 (#60): test_cutover_slug_rot's prose ratchet reports 58 hits against a ceiling of 54, so every PR current with main inherits the failure, #160 among them. Measured rather than inferred. The last green commit 8e6e7fa sits at 52 hits with 2 slots of slack, so #60 added SIX — not the four the ceiling arithmetic (58-54) suggests. c5ae1f8 (#96) added none. All six are in the new master-test-plan chapters, and the assertion's "New or changed" list is a tail slice of the sorted hits, not a diff, so it does not attribute them. Five are false positives in senses unrelated to the retired public mirror: three are `the mirror` matching INSIDE "the mirrorED" (the webview functions mirroring stepsModel, ADR 0108), one is a pip wheelhouse for air-gapped installs, and one is a detector spec quoting the very string it wants deleted — the _SELF collision this module already documents, reappearing in another file. The sixth is genuine rot. "the mirror-only gap" reincarnates the nightly cost lever that f7a337a (#25) swept from three files, leaving an explicit "do not re-add it" marker; windows-service-smoke is gated `github.repository == 'MEFORORG/MessageFoundry'`, i.e. ON this repo, which is the source. The line now names that gate instead of a mirror. The regex is deliberately NOT changed. Adding \b to `the mirror` clears three of the six and measures safe on the baseline (52 -> 52, suppressing only the three "mirrorED" lines) — but `private repo` -> "reporting" was a CROSS-LEMMA collision where \b removed noise, whereas `the mirror` -> "the mirrored" is the same lemma inflected, where it removes signal. Measured: \b blinds all five of "the mirrored workflows are rewritten by publish.ps1 on every push", "the mirroring job publishes ... nightly", "the mirrors are refreshed from the source repo each night", "CI runs on the source; the mirrored copy is read-only" and "the mirrored README is regenerated at release time", none of which carries a retrospective trigger — while both KEEP-class probes (the "mirror image of" idiom, a historical narrative) survive it untouched. Those verb forms are how a present-tense mirror claim actually gets written, so the boundary would preferentially blind the FIX class. Filing down the detector to clear a red is the failure this module exists to name. _PROSE_CEILING is unchanged at 54. The count returns to 52 — exactly the last-green posture — so no credit was earned to spend, and lowering it to 52 would set slack 0 as a new, stricter policy unrelated to this fix. Verified: 4/4 in the module; 52 hits / ceiling 54 / slack 2 across 1503 files; ruff check + ruff format clean; full suite 10222 passed, 836 skipped, with the sole failure test_gate_installed_parity — a local-machine test that skips on CI, whose installed-vs-source delta is one redacted comment line. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * chore(ide): bump fast-uri/brace-expansion/undici to patched versions (3 HIGH advisories) npm-audit (a required gate) began failing on every fresh CI run: three HIGH advisories now cover the ide/ transitive deps at their locked versions — fast-uri (GHSA-7p8r-x3mc-p8w7), undici (GHSA-8xcm-r25x-g524 + siblings), and brace-expansion (GHSA-rgw5-rvv9-x895). Each has an in-major patched release, so this is a lockfile bump, not a breaking major: fast-uri ^3.1.3 -> ^3.1.5 brace-expansion ^5.0.8 -> ^5.0.9 undici (new override) ^7.29.0 npm audit --package-lock-only now reports 0 vulnerabilities. Folded onto the main-unblock branch so one merge clears both the slug-rot test-red and npm-audit (the two were a circular deadlock). * fix(deps): bump cryptography 49->50 + pyOpenSSL 26.3->26.4 (CVE-2026-69247) pip-audit (a required gate) flagged cryptography 49.0.0 for CVE-2026-69247 — a PKCS7 decrypt Bleichenbacher/timing oracle, fixed in 50.0.0. The straight bump was gated by pyOpenSSL's cryptography<50 cap; pyOpenSSL 26.4.0 lifts it to <51, so both move together, within the existing pyproject floors (cryptography>=48.0.1, pyOpenSSL>=26.3.0) — no constraint change. Re-locked via uv and re-exported all six DEP-1 artifacts (uv lock --check clean); the stale pyproject <50 note -> <51. The advisory's primary oracle is already mitigated on the OpenSSL 3.2+ wheels this project ships; this clears the flagged version regardless. Third and final red gate on the main-unblock branch: with the slug-rot ratchet and the ide npm advisories, one merge now clears all three. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

wshallwshall added 3 commits July 28, 2026 14:10

Merge branch 'main' into claude/gate-liveness

9d96ba8

wshallwshall enabled auto-merge (squash) July 28, 2026 19:13

wshallwshall merged commit f7a337a into main Jul 28, 2026
33 checks passed

wshallwshall mentioned this pull request Jul 28, 2026

fix(ci): let the free-threaded canary report a problem — it was structurally unable to #27

Merged

wshallwshall mentioned this pull request Jul 28, 2026

probe: TEMPORARY diff-coverage probe — do not merge #28

Closed

wshallwshall deleted the claude/gate-liveness branch July 28, 2026 22:57

wshallwshall mentioned this pull request Jul 28, 2026

fix(ci): let the complexity delta account for itself — the receipt could not see it #31

Merged

Navigation Menu

Uh oh!

feat(ci): add a gate-liveness check — prove each quality gate actually measured something - #25

feat(ci): add a gate-liveness check — prove each quality gate actually measured something#25
wshallwshall merged 3 commits into
mainfrom
claude/gate-liveness

wshallwshall commented Jul 28, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

wshallwshall commented Jul 28, 2026

The distinction it rests on

Verified against real data, not fixtures

The adversarial review found this control carrying the same weakness it was built to catch

Also here

Verification

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant