Skip to content

test(parity): stop measuring this tool against itself - #45

Merged
blairham merged 1 commit into
mainfrom
fix-parity-harness
Aug 18, 2026
Merged

test(parity): stop measuring this tool against itself#45
blairham merged 1 commit into
mainfrom
fix-parity-harness

Conversation

@blairham

Copy link
Copy Markdown
Owner

The parity suite is the load-bearing claim of this project, and it was not measuring what it said.

What was wrong

pythonPreCommit() resolved the counterpart with exec.LookPath("pre-commit"). We install a binary called pre-commit — so on any machine that has this tool, LookPath finds it, and the suite diffs the tool against an older build of itself.

The committed report carries its own evidence: py="pre-commit 4.5.0 (build v4.5.3)". Real Python prints pre-commit 4.5.0 and nothing more; the (build …) suffix is ours. The advertised "96.2% parity, 3 failures" was v4.6.5 vs v4.5.3 of this project, and those three "differences from Python" were our own changes between two of our tags.

Two adjacent holes let a run measure nothing and still pass:

  • The file header documents -tags=integration, but there was no build tag, so make test ran the suite in the plain test job — where every check hits t.Skip. Run 31894464691 on main prints Total checks: 0 Pass: 0 Fail: 0 Parity: 0.0% and is green.
  • The parity job was gated on push or a test-languages label, so PRs did not run it.

What this does

  • Candidates must prove they are Python via --version ((build …) ⇒ this project, rejected by name), and every PATH entry is tried rather than the first hit, so having both installed resolves correctly.
  • The Python version is pinned to the parity line our version number claims (parityTarget = "4.6"). An off-target Python is refused rather than silently measured — v4.6.x means parity with 4.6.x, so measuring against 4.5 is measuring a claim we are not making.
  • The counterpart's version is recorded in the report. A parity percentage without the version it was taken against is not checkable.
  • PARITY_REQUIRE=1 (set in CI) turns "no counterpart" and "zero checks ran" into failures. //go:build integration keeps the suite out of make test, and the parity job now runs on every PR.
  • PARITY_PYTHON_PRE_COMMIT overrides the search, and fails loudly if it points at the wrong thing rather than falling back.

Verified, not assumed

Scenario Result
Counterpart = our own binary FAIL … (this project, not Python: "pre-commit 4.6.2 (build v4.6.2, …)")
Counterpart = Python 4.5.1 FAIL … (version 4.5.1 is not on the 4.6 line this project claims parity with)
No pre-commit on PATH, PARITY_REQUIRE=1 FAIL … Python pre-commit 4.6.x is required
No pre-commit on PATH, no PARITY_REQUIRE skips, and the report reads Measured against: NOTHING

The actual number

Run against genuine Python pre-commit 4.6.2, locally and in CI: 78/78, 100.0%. The broken harness was understating the tool. The regenerated report is committed and now names what it measured.

That number still only covers the CLI surface these 78 checks touch — see #43 for the language backends it does not reach yet.

Refs #43

The parity suite resolved its counterpart with LookPath("pre-commit").
This project installs a binary called pre-commit, so on any machine that
has it — every developer machine here — LookPath found *us*, and the
suite diffed the tool against an older build of itself.

The committed report is the proof: it records
py="pre-commit 4.5.0 (build v4.5.3)". Python prints "pre-commit 4.5.0"
and stops; the build suffix is ours. So the headline "96.2% parity, 3
failures" was v4.6.5 against v4.5.3, and the three differences were our
own changes between those tags, not divergence from Python.

Two more paths let a run measure nothing and still look healthy:

  - The header says -tags=integration but no build tag existed, so
    `make test` ran the suite in the ordinary test job, where every check
    hit t.Skip. That job printed "Total checks: 0 ... Parity: 0.0%" and
    passed green on every push to main.
  - The parity job itself was gated behind a push or a test-languages
    label, so PRs did not run it.

Now: candidates must prove they are Python by their --version output,
every PATH entry is tried rather than the first hit, the counterpart's
version is recorded in the report, and a run that resolves nothing or
compares nothing fails under PARITY_REQUIRE (set in CI). The Python
version is pinned to the line our own version number claims parity with,
and an off-target line is rejected by the harness rather than quietly
measured — parity against 4.5 is not the claim v4.6.x makes.

Measured for real, against Python pre-commit 4.6.2, the answer is
78/78. The suite was understating the tool, not flattering it.

Refs #43
@blairham
blairham merged commit 55f59c3 into main Aug 18, 2026
5 checks passed
@blairham
blairham deleted the fix-parity-harness branch August 18, 2026 01:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant