Skip to content

Fix the pre-existing astropy/pandas/shapely benchmark workflows - #7

Merged
atharvas merged 17 commits into
mainfrom
fix-benchmark-workflows
May 18, 2026
Merged

Fix the pre-existing astropy/pandas/shapely benchmark workflows#7
atharvas merged 17 commits into
mainfrom
fix-benchmark-workflows

Conversation

@atharvas

Copy link
Copy Markdown
Member

Summary

Repairs the three pre-existing, perpetually-red benchmark workflows (Test Astropy/Pandas/Shapely Benchmarks).

Three distinct root causes were found and fixed:

  1. Workflows auto-disabled. All three were disabled_inactivity (GitHub disables scheduled workflows after 60 days idle) — which is also why they never ran on PR SQLite snapshot storage, docs site, and release workflow #6. Re-enabled via gh workflow enable; this PR triggers them.

  2. py3.8 collection bug. tests/test_cli_roundtrip.py used list[Path] annotations without from __future__ import annotations, raising TypeError: 'type' object is not subscriptable at collection time on Python 3.8 — erroring out the py3.8 leg of all three workflows (and violating the repo's own stated convention). Added the future import.

  3. equal_nan=False flagged deterministic NaNs as changes. The dominant real failure class (e.g. stats.sigma_clipping.*) was Arrays not close: max_diff=nan: benchmarks that deterministically return NaN-containing arrays "failed" because the comparator defaulted to equal_nan=False. For snapshot testing, a NaN that reappears unchanged is not a regression. Changed ComparisonConfig / SnapshotConfig / CLI fallbacks to default equal_nan=True (equal_nan=False still available for strict numpy.isclose semantics). Behavior change → version bumped 0.2.0 → 0.2.1.

  4. Roundtrip gate is now regression-based, not perfection-based. Real third-party suites contain inherently un-snapshotable benchmarks (memory addresses in reprs, timing-sensitive, dtype-unstable) that can never pass a "zero failures" assertion. test_cli_roundtrip.py now runs list → capture → baseline → verify and asserts the baseline→verify transition matrix has no pass-to-fail / skip-to-fail (read from verify --summary). Consistently-broken benchmarks stay fail-to-fail and are tolerated; genuine regressions are still caught. This uses the tool's own baseline subcommand + transition matrix.

Local validation

Suite Before After
shapely roundtrip (both tests) py3.8 collection error PASS (82s)
astropy core shard 127 / 114 pass / 8 fail PASS — 0 regressions (13.5 min)
fast unit suite 155 passed, 8 skipped
ruff check + ruff format --check clean
comparator tests (incl. new NaN coverage) 60 passed

Test plan

  • CI: tests (py3.8–3.13) + ruff
  • CI: Test Shapely/Astropy/Pandas Benchmarks (re-enabled) go green on this PR

@atharvas
atharvas merged commit f08bb0c into main May 18, 2026
30 of 49 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant