Fix the pre-existing astropy/pandas/shapely benchmark workflows - #7
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Repairs the three pre-existing, perpetually-red benchmark workflows (
Test Astropy/Pandas/Shapely Benchmarks).Three distinct root causes were found and fixed:
Workflows auto-disabled. All three were
disabled_inactivity(GitHub disables scheduled workflows after 60 days idle) — which is also why they never ran on PR SQLite snapshot storage, docs site, and release workflow #6. Re-enabled viagh workflow enable; this PR triggers them.py3.8 collection bug.
tests/test_cli_roundtrip.pyusedlist[Path]annotations withoutfrom __future__ import annotations, raisingTypeError: 'type' object is not subscriptableat collection time on Python 3.8 — erroring out the py3.8 leg of all three workflows (and violating the repo's own stated convention). Added the future import.equal_nan=Falseflagged deterministic NaNs as changes. The dominant real failure class (e.g.stats.sigma_clipping.*) wasArrays not close: max_diff=nan: benchmarks that deterministically return NaN-containing arrays "failed" because the comparator defaulted toequal_nan=False. For snapshot testing, a NaN that reappears unchanged is not a regression. ChangedComparisonConfig/SnapshotConfig/ CLI fallbacks to defaultequal_nan=True(equal_nan=Falsestill available for strictnumpy.isclosesemantics). Behavior change → version bumped 0.2.0 → 0.2.1.Roundtrip gate is now regression-based, not perfection-based. Real third-party suites contain inherently un-snapshotable benchmarks (memory addresses in reprs, timing-sensitive, dtype-unstable) that can never pass a "zero failures" assertion.
test_cli_roundtrip.pynow runslist → capture → baseline → verifyand asserts the baseline→verify transition matrix has nopass-to-fail/skip-to-fail(read fromverify --summary). Consistently-broken benchmarks stayfail-to-failand are tolerated; genuine regressions are still caught. This uses the tool's ownbaselinesubcommand + transition matrix.Local validation
coreshardruff check+ruff format --checkTest plan
tests(py3.8–3.13) +ruffTest Shapely/Astropy/Pandas Benchmarks(re-enabled) go green on this PR