Unify the versioning scheme - #17
Merged
Merged
Conversation
The repo carried three version identifiers in two styles, with nothing stating how they relate: `bench_version` (`2026.08.17-v0_4.2`), `site.site_version` (`2026.07.10-fixtures`, read by no code at all), and the harness `0.1.0`. The `v0_4.x` part is lineage from a private repository and carries no meaning here. Two axes, one style. `bench_version` becomes plain semver and its shape is now enforced by `validate`, so the number answers the only question it exists for: whether two runs are comparable. MAJOR is task membership, graders, fixtures or capability assignments; MINOR is additive; PATCH is descriptions and other non-executable metadata. The published runs sit at `0.4.1` and HEAD at `0.4.2`, which under these rules is a PATCH apart — hence the numbering rather than a fresh `1.0.0`, which would have read as an incompatible dataset. The fixture tree belongs to the dataset axis, so `site_version` is dropped rather than kept in sync; a run already records `runner.fixtures.tree_sha256`, which is checkable where a hand-maintained label is not. The harness version moves to `runner/version.py` as the single source `pyproject.toml` reads, and runs now record it as `harness_version`. Reports keep regenerating byte-identically from the published evidence: runs predating this scheme carry no `harness_version`, and the report header only gains the field when one is present.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #16.
The problem
Three version numbers, no stated relationship between them:
manifest.json→bench_version2026.08.17-v0_4.2manifest.json→site.site_version2026.07.10-fixturespackage.json,pyproject.toml0.1.0Two of them are dates, one is semver, and the
v0_4.xpart is left over from the private repo.What this PR does
Keeps two of them, deletes the third.
1. The dataset version becomes
0.4.2(was2026.08.17-v0_4.2).This number answers one question: if I re-run the benchmark, can I compare my numbers with the published ones? So it is bumped by what the answer becomes:
validatenow rejects anything that is notMAJOR.MINOR.PATCH.Why
0.4.2and not1.0.0: the published runs recorded2026.08.02-v0_4.1, which is0.4.1. Everything that changed since is task descriptions — a PATCH. Restarting at1.0.0would tell a reader the dataset changed and the published numbers no longer apply, which is not true.2. The harness version stays
0.1.0, but now lives in one place.runner/version.pyholds it,pyproject.tomlreads it from there, and a test keepspackage.jsonin sync. Runs record it asharness_version, so a report can say which code produced it.3.
site_versionis deleted.Nothing read it. The fixtures it labelled are part of the dataset: editing a fixture changes results, so that is a MAJOR bump of the dataset version. Every run already records
runner.fixtures.tree_sha256, which can be checked — unlike a label someone has to remember to update.What breaks
bench_versionis now a hardvalidateerror. The three published runs keep their old string; new runs record semver.run_manifest.jsonno longer carriessite.site_version. It is in the three published run manifests and will be in none of the future ones.Neither affects reproducibility: a run is identified by the sha256 digests in
run_manifest.json, not by these labels.Checks
validate,scenarios --check,pytest: 599 passed, 7 skipped (11 new)