test(parity): stop measuring this tool against itself - #45
Merged
Conversation
The parity suite resolved its counterpart with LookPath("pre-commit").
This project installs a binary called pre-commit, so on any machine that
has it — every developer machine here — LookPath found *us*, and the
suite diffed the tool against an older build of itself.
The committed report is the proof: it records
py="pre-commit 4.5.0 (build v4.5.3)". Python prints "pre-commit 4.5.0"
and stops; the build suffix is ours. So the headline "96.2% parity, 3
failures" was v4.6.5 against v4.5.3, and the three differences were our
own changes between those tags, not divergence from Python.
Two more paths let a run measure nothing and still look healthy:
- The header says -tags=integration but no build tag existed, so
`make test` ran the suite in the ordinary test job, where every check
hit t.Skip. That job printed "Total checks: 0 ... Parity: 0.0%" and
passed green on every push to main.
- The parity job itself was gated behind a push or a test-languages
label, so PRs did not run it.
Now: candidates must prove they are Python by their --version output,
every PATH entry is tried rather than the first hit, the counterpart's
version is recorded in the report, and a run that resolves nothing or
compares nothing fails under PARITY_REQUIRE (set in CI). The Python
version is pinned to the line our own version number claims parity with,
and an off-target line is rejected by the harness rather than quietly
measured — parity against 4.5 is not the claim v4.6.x makes.
Measured for real, against Python pre-commit 4.6.2, the answer is
78/78. The suite was understating the tool, not flattering it.
Refs #43
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The parity suite is the load-bearing claim of this project, and it was not measuring what it said.
What was wrong
pythonPreCommit()resolved the counterpart withexec.LookPath("pre-commit"). We install a binary calledpre-commit— so on any machine that has this tool, LookPath finds it, and the suite diffs the tool against an older build of itself.The committed report carries its own evidence:
py="pre-commit 4.5.0 (build v4.5.3)". Real Python printspre-commit 4.5.0and nothing more; the(build …)suffix is ours. The advertised "96.2% parity, 3 failures" was v4.6.5 vs v4.5.3 of this project, and those three "differences from Python" were our own changes between two of our tags.Two adjacent holes let a run measure nothing and still pass:
-tags=integration, but there was no build tag, somake testran the suite in the plaintestjob — where every check hitst.Skip. Run 31894464691 on main printsTotal checks: 0 Pass: 0 Fail: 0 Parity: 0.0%and is green.pushor atest-languageslabel, so PRs did not run it.What this does
--version((build …)⇒ this project, rejected by name), and every PATH entry is tried rather than the first hit, so having both installed resolves correctly.parityTarget = "4.6"). An off-target Python is refused rather than silently measured —v4.6.xmeans parity with 4.6.x, so measuring against 4.5 is measuring a claim we are not making.PARITY_REQUIRE=1(set in CI) turns "no counterpart" and "zero checks ran" into failures.//go:build integrationkeeps the suite out ofmake test, and the parity job now runs on every PR.PARITY_PYTHON_PRE_COMMIToverrides the search, and fails loudly if it points at the wrong thing rather than falling back.Verified, not assumed
FAIL … (this project, not Python: "pre-commit 4.6.2 (build v4.6.2, …)")FAIL … (version 4.5.1 is not on the 4.6 line this project claims parity with)PARITY_REQUIRE=1FAIL … Python pre-commit 4.6.x is requiredPARITY_REQUIREMeasured against: NOTHINGThe actual number
Run against genuine Python pre-commit 4.6.2, locally and in CI: 78/78, 100.0%. The broken harness was understating the tool. The regenerated report is committed and now names what it measured.
That number still only covers the CLI surface these 78 checks touch — see #43 for the language backends it does not reach yet.
Refs #43