Skip to content

279 tidy asv docs and setup #280

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
alexallmont wants to merge 5 commits into
base: 220-roughpy-jax
Choose a base branch
from
Open

279 tidy asv docs and setup #280

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
463a448
docs: remove out of date asv instructions (#279)
alexallmont Jan 19, 2026
a4c0bdb
fix: remove asv branch specifier (#279)
alexallmont Jan 19, 2026
323ab6f
tidy: gitignore asv-generated files (#279)
alexallmont Jan 19, 2026
c596744
docs: correct branch in ASV docs (#279)
alexallmont Jan 20, 2026
d8cc8f3
tidy: json format asv.conf (#279)
alexallmont Jan 20, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -375,4 +375,6 @@ examples/*.png
examples/documentation-testing.py

todo.md
tools/vcpkg/
tools/vcpkg/

.asv/

@jackleland jackleland Jan 22, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not 100% sure on best practice here, we might want to store the results of benchmarks in the repo, but definitely not teh environment it creates for benchmarking in.

39 changes: 23 additions & 16 deletions asv.conf.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,48 +4,55 @@
"project_url": "https://github.com/datasig-ac-uk/RoughPy",
"repo": ".",
"branches": [
"jl/benchmarks"
"<asv.conf set your branch name>"

@inakleinbottle inakleinbottle Jan 21, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this not a placeholder? Surely the default branch for running benchmarks is the main branch?

@alexallmont alexallmont Jan 21, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a placeholder with an intentionally ugly/greppable name, so it's easy to find if you run asv without having read the README. If you do so, you will see this.

> asv run --launch-method spawn
· Unknown branch <asv.conf set your branch name> in configuration

Longer term, we should be able to get rid of branches in the json, but for now it needs to be set to your working branch name otherwise, asv complains that roughpy_jax needs to be pip-installed, i.e. it does not pick up the local roughpy_jax subdir unless this is set.

@jackleland and I hope that this will be resolved after we have got the packaging working.

@jackleland jackleland Jan 21, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To be clear, it should be set to main (or whichever branch we want to run benchmarks on long-term) but removing it defaults to main.

What confuses me is that this should have worked but built the contents of the roughpy_jax folder on that branch, rather than not building it at all, but I'm not sure we need to worry about this too much

@inakleinbottle inakleinbottle Jan 22, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we're writing an action for testing performance regression it is probably best to override with whatever git head initiated the action, be that a push to main branch or a PR into main or whatever. To be honest I don't really understand the execution model for ASV. It seems that one should run it on the code as it exists in the repo that is cloned. Cloning a particular branch other than this seems self-defeating.

Also, I think this is complicated by the fact that we really have two projects in the same repo that are somewhat unrelated. I'm seeing yet more evidence for separating the roughpy_jax into a separate repo.

],
"dvcs": "git",
"environment_type": "virtualenv",
"show_commit_url": "https://github.com/datasig-ac-uk/RoughPy/commit/",
"pythons": ["3.12"],

"pythons": [
"3.12"
],
"matrix": {
"jax": ["0.8.0"],
"jaxlib": ["0.8.0"],
"numpy": ["2.3.4"]
"jax": [
"0.8.0"
],
"jaxlib": [
"0.8.0"
],
"numpy": [
"2.3.4"
]
},

"exclude": [
{"python": "3.12", "jax": null}
{
"python": "3.12",
"jax": null
}
],

"benchmark_dir": "benchmarks/",
"env_dir": ".asv/env",
"results_dir": ".asv/results",
"html_dir": ".asv/html",

"install_command": [
"python -m pip install {wheel_file}"
],

"uninstall_command": [
"return-code=any python -m pip uninstall -y {project}"
],

"build_command": [
"python -m pip install scikit-build-core[pyproject] 'pybind11<3.0.0' numpy hatch-fancy-pypi-readme jax",
"CMAKE_ARGS='-DROUGHPY_JAX=ON' python -m pip wheel --no-build-isolation --no-deps -vvv {build_dir} -w {build_cache_dir}"
],

"benchmark_pattern": "^(Bench|Track|Suite)",
"regressions_first_commits": {},
"regressions_thresholds": {
"factors": {"factor": 2.0},
"percentages": {"percentage": 20.0}
"factors": {
"factor": 2.0
},
"percentages": {
"percentage": 20.0
}
},

"hash_length": 8,
"cpu_count": null,
"build_cache_size": 1
Expand Down
18 changes: 11 additions & 7 deletions benchmarks/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,10 +19,18 @@ This installs:
### Running Benchmarks

#### Initial Setup
```bash
# Navigate to benchmarks directory
cd benchmarks

Open `asv.conf.json` and set your branch name, i.e. replace with result of `git branch --show-current`:

```json
"branches": [
"<asv.conf set your branch name>"
],
```

Then in the shell:

```bash
# Initialize ASV (first time only)
asv machine --yes

Expand All @@ -32,8 +40,6 @@ asv run --launch-method spawn

#### Continuous Use
```bash
# From benchmarks directory:

# Run benchmarks for specific commits
asv run main^..HEAD

Expand All @@ -47,8 +53,6 @@ asv preview

#### Development Testing
```bash
# From benchmarks directory:

# Quick test of benchmark validity
asv dev

Expand Down