Skip to content

gh-995: Allow filtering a single benchmark test #1068

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
connoraird merged 2 commits into from
May 12, 2026
Merged

gh-995: Allow filtering a single benchmark test #1068

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
d383a3b
Allow passing pytest flags through to benchmarks and regression test …
connoraird May 12, 2026
8b3f624
Document passing pytest args to benchmarks
connoraird May 12, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 18 additions & 4 deletions CONTRIBUTING.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -211,8 +211,8 @@ supported Python versions using -
uv run nox -s tests
```

Only `tests`, `coverage`, `coverage_benchmarks`, `doctests`, `benchmarks`, and
the `regression_tests` session run on all supported Python versions by default.
Only `tests`, `coverage` and the `doctests` session run on all supported Python
versions by default.

To specify a particular Python version (for example `3.14`), use the following
syntax -
Expand Down Expand Up @@ -253,10 +253,11 @@ To ensure that _GLASS_ remains performant over time, a set of benchmarks are
provided in the [benchmarks](./tests/benchmarks/) folder. These benchmarks can
be run through nox.

A single benchmark can be run by specifying the revision to benchmark.
All of the benchmarks for a given revision can be run by specifying the revision
to benchmark.

```sh
uv run nox -s benchmark -- <revision-to-benchmark>
uv run nox -s benchmarks -- <revision-to-benchmark>
```

The benchmarks can be used to run a regression test of _GLASS_. These regression
Expand All @@ -267,6 +268,19 @@ _GLASS_.
uv run nox -s regression-tests -- <initial-state-revision> <revision-to-compare>
```

To filter the benchmark tests to be ran, one can pass pytest arguments via the
cli after the required revision arguments. For example, if you wished to run
only benchmarks from the `test_fields.py` file you could run one of the
following commands:

```sh
# Benchmarks
uv run nox -s benchmarks -- <revision-to-benchmark> -k test_fields
# Regression tests
uv run nox -s regression-tests -- <initial-state-revision> \
<revision-to-compare> -k test_fields
```

<!-- prettier-ignore -->
> [!TIP]
> Benchmark tests should do minimal assertions on what is returned to
Expand Down
14 changes: 11 additions & 3 deletions noxfile.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,10 +64,10 @@ def _check_revision_count(
msg = f"{expected_count} revision(s) not provided"
raise ValueError(msg)

if len(session_posargs) != expected_count:
if len(session_posargs) < expected_count:
msg = (
f"Incorrect number of revisions provided ({len(session_posargs)}), "
f"expected {expected_count}"
f"expected at least {expected_count}"
)
raise ValueError(msg)

Expand Down Expand Up @@ -239,7 +239,12 @@ def benchmarks(session: nox.Session) -> None:

# overwrite current package with specified revision
session.install(f"git+{GLASS_REPO_URL}@{revision}")
session.run("pytest", BENCH_TESTS_LOC)
session.run(
"pytest",
BENCH_TESTS_LOC,
*SHARED_BENCHMARK_FLAGS,
Copy link
Copy Markdown
Member

@paddyroddy paddyroddy May 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why weren't we using these before?

Copy link
Copy Markdown
Contributor Author

@connoraird connoraird May 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we just missed this.

*session.posargs[1:],
)


@nox_uv.session(
Expand Down Expand Up @@ -271,6 +276,7 @@ def regression_tests(session: nox.Session) -> None:
BENCH_TESTS_LOC,
"--benchmark-autosave",
*SHARED_BENCHMARK_FLAGS,
*session.posargs[2:],
)

session.log(f"Comparing {before_revision} benchmark to revision {after_revision}")
Expand All @@ -284,6 +290,7 @@ def regression_tests(session: nox.Session) -> None:
"--benchmark-compare=0001",
"--benchmark-compare-fail=mean:5%",
*SHARED_BENCHMARK_FLAGS,
*session.posargs[2:],
)

session.log("Running unstable regression tests")
Expand All @@ -296,4 +303,5 @@ def regression_tests(session: nox.Session) -> None:
# Absolute time comparison in seconds
"--benchmark-compare-fail=mean:0.0005",
*SHARED_BENCHMARK_FLAGS,
*session.posargs[2:],
)
Loading