Skip to content

fix: make Specifier / SpecifierSet pickle-safe#1168

Open
henryiii wants to merge 7 commits intopypa:mainfrom
henryiii:henryiii/fix/specset
Open

fix: make Specifier / SpecifierSet pickle-safe#1168
henryiii wants to merge 7 commits intopypa:mainfrom
henryiii:henryiii/fix/specset

Conversation

@henryiii
Copy link
Copy Markdown
Contributor

This is a followup to #1163. I've had an model apply the same changes/logic in that PR to Specifier/SpecifierSet. I think this is likely in better shape, due to not referncing a missing module, but should be better long term.

I also had it generate similar tests for several packaging versions.

🤖 Assisted by both OpenCode w/ Kimi-K2.5 & Copilot CLI w/ (new for CLI) auto model selection (claude-sonnet-4.6).

CC @eachimei.

@henryiii henryiii force-pushed the henryiii/fix/specset branch from 8e44fef to 0a5742b Compare April 20, 2026 04:08
@eachimei
Copy link
Copy Markdown
Contributor

Looks good! 👍
One observation: the indexed-tuple approach optimizes for compactness and performance, at the cost of relying on shape-sniffing (isinstance, len, nested type checks) to distinguish formats in __setstate__. That's perfectly reasonable for this and Version use cases, just noting that there's no explicit format tag or version marker, so future format changes will need another heuristic branch rather than a clean dispatch. Not a current risk, and if it does become a problem in the future, a format tag can then be introduced at that point...

@henryiii
Copy link
Copy Markdown
Contributor Author

Yes, I usually start the tuple with an "format version" integer. But I don't think this will be likely to change much, and simply adding a value is safe. Longer term, we could probably use pattern matching here if it's not too much slower.

@eachimei
Copy link
Copy Markdown
Contributor

Indeed, sounds good

@henryiii henryiii force-pushed the henryiii/fix/specset branch from 3954b39 to 3c2603d Compare April 20, 2026 12:55
@henryiii henryiii marked this pull request as ready for review April 20, 2026 12:59
@henryiii henryiii force-pushed the henryiii/fix/specset branch 2 times, most recently from a56da7e to 3799a8a Compare April 20, 2026 16:30
henryiii and others added 6 commits April 21, 2026 10:22
Signed-off-by: Henry Schreiner <henryfs@princeton.edu>
Assisted-by: Copilot:claude-sonnet-4.6
Signed-off-by: Henry Schreiner <henryfs@princeton.edu>
Assisted-by: OpenCode:Kimi-K2.5
Signed-off-by: Henry Schreiner <henryfs@princeton.edu>
Signed-off-by: Henry Schreiner <henryfs@princeton.edu>
Assisted-by: Copilot:claude-sonnet-4.6
Signed-off-by: Henry Schreiner <henryfs@princeton.edu>
Signed-off-by: Henry Schreiner <henryfs@princeton.edu>
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR extends the pickle-safety/backward-compatibility work from #1163 to Specifier and SpecifierSet, aiming to ensure pickles remain loadable across packaging versions and don’t persist cache/internal state.

Changes:

  • Add __getstate__/__setstate__ to Specifier and SpecifierSet, using compact tuple state and discarding caches on restore while supporting older pickle formats.
  • Add pickle roundtrip tests for Specifier/SpecifierSet.
  • Add fixture-based tests to load historical pickle payloads (packaging 25.0, 26.0, 26.1, 26.2-style).

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File Description
src/packaging/specifiers.py Implements compact pickle state + legacy restore logic for Specifier and SpecifierSet, clearing caches on restore.
tests/test_specifiers.py Adds roundtrip and historical-pickle loading tests for specifiers/specifiersets across multiple packaging versions.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/packaging/specifiers.py Outdated
Comment thread src/packaging/specifiers.py Outdated
Comment thread tests/test_specifiers.py
@henryiii
Copy link
Copy Markdown
Contributor Author

henryiii commented Apr 21, 2026

The copilot review boils down to one thing: protect against malformed pickles. I've added it (generated tests, but threw away the model-implemented solution and added my own), but I'm not sure it's really needed to always check this (auto-generated pickle code doesn't have such checks), but I think it's probably a good idea for the "old" pickles. I've added it for everything, but I'd also be fine to drop it for new-style pickles (outer tuple). It's not too expensive, though, and it does make typing happy.

Pattern matching would be so nice here. :)

Signed-off-by: Henry Schreiner <henryfs@princeton.edu>
@henryiii henryiii force-pushed the henryiii/fix/specset branch from 47b00d5 to cab5c2c Compare April 21, 2026 15:45
@henryiii
Copy link
Copy Markdown
Contributor Author

henryiii commented Apr 21, 2026

Also, I've followed #1163 in generating old pickles and storing them, but we could also just store the intermediate state, which wouldn't be a somewhat opaque binary string, and would unit test the same. I think the current one is a bit better, but if there's a desire to avoid binary strings, we could make something work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants