Skip to content

fix(benchmarks): key history by build profile and admit dirty native builds - #838

Merged
jtravs merged 1 commit into
mainfrom
fix/bench-history-138
Nov 15, 2026
Merged

jtravs merged 1 commit into
mainfrom
fix/bench-history-138

Conversation

@jtravs

@jtravs jtravs commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

The comparability key holds the build profile alone, as #138 recommended and the maintainer confirmed. With it, a compiler or BLAS change inside the wheel still compares, while a local build does not.

Part 1, the key.

  • benchmarks/history.py comparable() now includes the build profile: the first ; field of the existing native_build column (distribution, local or reference).
  • The key is derived from data already stored. The CSV format and columns are unchanged.
  • Each committed history file holds one profile, so existing comparisons are unchanged.

Part 2, dirty builds.

  • benchmarks/track.py gains native_build_commit(), which strips +dirty, and passes --allow-dirty to imported_elsewhere.
    • With the flag, a dirty build is admitted, its rows are marked dirty=1 with the bare native_commit, and the staleness check still applies.
    • Without the flag, the run is refused, and the message names the modified tree instead of advising a rebuild that would be stamped +dirty again.
  • native/CMakeLists.txt's dirty check now excludes benchmarks/history. A probe under cmake -P gave:
    • clean tree → 0;
    • only a history CSV modified → 0;
    • a native/ file modified too → 1.
  • docs/dev/benchmarking.md is updated to match.

Tests: two regression tests in tests/test_benchmark_track.py, using the issue's reproduction (shared_previous 0 across profiles) and the dirty admit/refuse cases. 34 passed.

Still open in #138: the "page says one thing" item (CPU rows time the wheel), and the "Also" items: misaligned rows from history.append, and the stale version label.

Refs #138

🤖 Generated with Claude Code

https://claude.ai/code/session_01XGENPwwkUUG9KSAwneHjgH

…builds

Rows now compare only within one native build profile (the first field of
native_build), so a local build after the release's wheel starts a new
series instead of reading as a speed-up; a toolchain change inside the
wheel still compares. imported_elsewhere strips the build's +dirty stamp
before asking git, admits it with --allow-dirty (marking the rows dirty)
and otherwise refuses by naming the modified tree. CMake's dirty check
ignores benchmarks/history, which one checkout may modify to record
several runs.

Refs #138

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XGENPwwkUUG9KSAwneHjgH
@read-the-docs-community

Copy link
Copy Markdown

Documentation build overview

📚 solprop | 🛠️ Build #35891755 | 📁 Comparing 398528d against latest (07218ed)

  🔍 Preview build  

1 file changed
± dev/benchmarking.html

@codecov

codecov Bot commented Nov 14, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@jtravs
jtravs marked this pull request as ready for review November 15, 2026 17:55
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-15T17:56:19.552070Z 398528d Draft marked ready
🔒 Security Review ✅ Completed 2026-11-15T17:58:38.131488Z 398528d Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@jtravs
jtravs merged commit 743dd4f into main Nov 15, 2026
72 checks passed
@jtravs
jtravs deleted the fix/bench-history-138 branch November 15, 2026 20:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant