Repository navigation
Description
What happens
A build change reads as a speed change. comparable keys rows by backend, width, case and settings, not by native build. The 0.1.0 CPU rows time the release's wheel, while the same page's admission list says to run from uv sync --extra native, which is a local build. The page's first commands say uv run, which removes the wheel, and the refusal says to rebuild. Following those at the next release passes every check and appends to the same file, and python -m benchmarks history reports the build's gain as the release's.
--allow-dirty does not admit a dirty build. CMake stamps +dirty when git diff --quiet HEAD finds anything modified in the tree, including benchmarks/history/*.csv, which one checkout may leave modified to record several runs. imported_elsewhere passes <commit>+dirty to git diff, which cannot resolve it. So the run is refused, after it has finished, with the wrong cause, and rebuilding stamps +dirty again. --allow-dirty gates only the source check, not this one. Whether a rebuild follows a recorded run in practice is not known: the wheel recipe never rebuilds.
Why it matters
The history is the project's record of speed between releases, and its first comparison is the next release's. A refused run costs a whole run, about an hour per machine, though its measurements survive in the bundle.
Reproduction
From the repository root:
import subprocess
import sys
import tempfile
from pathlib import Path
sys.path.insert(0, ".") # from the repository root
from benchmarks import history
from benchmarks.system import native_changed
from benchmarks.track import imported_elsewhere
base = {name: "" for name in history.COLUMNS}
base.update(date="2026-11-01T08:00Z", solprop="0.1.0", commit="a" * 12, dirty=0, backend="native", slug="x",
threads=1, threading="passive", numa_domains=1, case="rdw", revision=1, stepper="default")
first = {**base, "native_build": "distribution; GCC; FFTW_OPENMP; OpenBLAS", "median_s": "10.0"}
second = {**base, "date": "2026-11-02T08:00Z", "commit": "b" * 12, "native_build": "local; GCC; FFTW_OPENMP; MKL", "median_s": "7.0"}
path = Path(tempfile.mkdtemp()) / "build.csv"
history.append(path, [first, second])
for row in history.summary(history.read(path)):
print("summary", row["date"], "shared with the previous", row["shared_previous"], "ratio", row["ratio_previous"])
revision = subprocess.check_output(["git", "rev-parse", "HEAD"], text=True).strip()
print("native_changed, same commit:", native_changed(Path("."), revision, revision))
print("native_changed, the commit +dirty:", native_changed(Path("."), revision + "+dirty", revision))
print(imported_elsewhere([], {"build_info": {"git_commit": revision + "+dirty"}}, {"revision": revision}))summary 2026-11-01T08:00Z shared with the previous 1 ratio 1.0
summary 2026-11-02T08:00Z shared with the previous 1 ratio 0.7
native_changed, same commit: False
native_changed, the commit +dirty: True
native/ changed since the native backend's build (9d5d92f678f0); rebuild it (uv sync --extra native)
Context
- Benchmarks: 0.1.0 performance history (native + numba), any-backend history #68 made this the record, and added the history files' exception in the harness only.
comparableis from SEP 38. Benchmark harness: duplicated helpers, coverage and a conflated result #106 covers other harness items.
To do
- Add the build profile, the first field of
native_build, tocomparable. - Make the page say one thing: CPU rows time the wheel, as line 182 has it, and GPU rows a local build. That touches line 12, line 141 and the refusal's advice.
- Pass
--allow-dirtytoimported_elsewhere. Strip and report+dirtyseparately: with the flag accept it and mark the rows dirty, otherwise refuse by naming a build from a modified tree. - Exclude
benchmarks/historyfrom CMake's dirty check.
Needs a decision before work starts: what the key holds. Recommended: the profile alone. A compiler or BLAS change inside the wheel is part of what a release ships, so it should still compare, while a local build is a different thing. The alternative, the whole field, would stop every comparison across a toolchain update.
Done when
In the reproduction the two builds' rows no longer compare (shared_previous 0). imported_elsewhere with allow_dirty=True admits a +dirty build and marks its rows, and without it refuses naming a modified tree. A modified benchmarks/history file stamps no +dirty. The page says one thing. Tests hold the first two.
Checked against commit 9d5d92f; the code links are pinned to it.
Also
append can write misaligned rows. history.append rewrites the file only when it holds rows under other columns, so an empty file gets rows with no header, a header-only file of other columns gets rows under that header, and a file with no trailing newline runs the first new row onto its last line. No normal path is known to reach these.
import csv, sys, tempfile
from pathlib import Path
sys.path.insert(0, ".") # from the repository root
from benchmarks import history
row = {name: "" for name in history.COLUMNS}
row.update(date="2026-11-01T08:00Z", solprop="0.1.0", case="rdw", median_s="1.0")
folder = Path(tempfile.mkdtemp())
newline_less = folder / "a.csv"
history.append(newline_less, [row])
newline_less.write_text(newline_less.read_text().rstrip("\n"))
history.append(newline_less, [{**row, "case": "gnlse_raman_shock"}])
rows = list(csv.DictReader(newline_less.open()))
print("no trailing newline:", len(rows), "row, its case", rows[0]["case"], "and overflow", len(rows[0].get(None, [])))
empty = folder / "b.csv"
empty.touch()
history.append(empty, [row])
print("empty file: first line starts", empty.read_text().splitlines()[0][:30])
old = folder / "c.csv"
old.write_text("date,solprop\n")
history.append(old, [row])
rows = list(csv.DictReader(old.open()))
print("header-only, other columns: fieldnames", list(rows[0])[:2], "overflow", len(rows[0].get(None, [])))no trailing newline: 1 row, its case rdw and overflow 34
empty file: first line starts 2026-11-01T08:00Z,0.1.0,,,,,,,
header-only, other columns: fieldnames ['date', 'solprop'] overflow 33
A stale version label. The solprop column comes from dist-info, so a later checkout in the same venv keeps the old label: the venv here reports 0.0.1.dev444+gcc6d8e2b0 on this commit, where git describe gives v0.1.0-55-g8a9f764. This affects the label only, since the commit column holds the revision.
- Make
appendrewrite whenever the file does not end in the expected header and a newline, and test the three forms above. - Read the version from the checkout, or say in
benchmarking.mdthatcommitis the record andsolproponly a label.
Added on 2026-11-01, checked against commit 8a9f764.
Activity
jtravs commentedon Nov 14, 2026
Picked up by the issue-fixing session; a PR will reference this issue.
jtravs commentedon Nov 15, 2026
#838 (merged) fixes the comparability key. History rows now compare by build profile alone (distribution, local or reference), as the maintainer decided. It also lets
--allow-dirtyadmit a dirty native build. Still open here: the "page says one thing" item (the CPU rows time the wheel), and the "Also" items: misaligned rows fromhistory.append, and the stale version label.