Skip to content

Faster pickling and from_json; stop mutating inputs; bump to 1.2.0 - #20

Merged
iskandr merged 1 commit into
masterfrom
faster-pickle-and-decode
Oct 6, 2026
Merged

iskandr merged 1 commit into
masterfrom
faster-pickle-and-decode

Conversation

@iskandr

@iskandr iskandr commented Oct 6, 2026 •

Copy link
Copy Markdown
Collaborator

Stacked on #18, which is stacked on #19. Bases retarget automatically as each PR merges.

Changes

  • Pickle via from_dict(to_dict()). __reduce__ (on both Serializable and DataclassSerializable) now returns (type(self).from_dict, (self.to_dict(),)) instead of converting the whole object tree to its JSON-shaped representation first. Values in to_dict() are pickled natively, so e.g. a datetime field now pickles (it never could before).
    • Pickles written by earlier versions still load: they call from_serializable_repr, which is unchanged (tested by re-creating the pre-1.2 format with a reducer_override pickler).
    • Pickles written by this version load with earlier versions too: they just reference Cls.from_dict.
    • Singletons returned by from_dict (pyensembl's EnsemblRelease/Species) still unpickle to the same object.
    • copy.copy() now gives a true shallow copy; it used to rebuild nested objects. No downstream code copies these objects.
  • Single-pass from_json. Objects are rebuilt bottom-up in a json.loads object_hook instead of a second walk over the parsed tree. A test checks the result matches from_serializable_repr(json.loads(...)).
  • Bug: from_serializable_repr damaged its input. It popped keys out of the dicts it was given, so decoding the same representation twice raised IndexError. It no longer modifies its input.
  • Bug: Serializable.from_dict modified the caller's dict. It renamed aliased keys in place; it now copies first.
  • Per-class cache of __init__ argument names. This lookup runs on every default to_dict/__eq__/__hash__. Also drops the __new__ fallback, which could never succeed on Python 3.
  • Documented and tested: keyword-only __init__ args are left out of the default to_dict(). varcode's SpliceOutcomeSet depends on this.
  • Docs from Add reader-oriented docs and executable JSON examples (1.1.1) #19 updated. The README and docs/guides/objects.md now say the default to_dict() uses positional args only, and the guide has a new "Pickle objects" section with a checked example. scripts/check_docs_examples.py runs all 6 examples, and ./docs.sh (strict build plus link check) passes.

The JSON wire format is unchanged: output is byte-identical against golden payloads (nested objects, sets, tuples, namedtuples, non-string keys, functions, classes, OrderedDict, dataclasses, unicode, extreme floats).

Benchmarks

5k varcode-style Variants, best of 5, Python 3.12, compared with 1.1.1:

1.1.1 this PR
pickle.dumps 19.6 ms 6.4 ms
pickle.loads 29.1 ms 6.5 ms
pickle size 0.38 MB 0.24 MB
from_json 34.3 ms 21.8 ms
to_json 28.9 ms 27.2 ms
hash / == 1.55 / 0.95 µs 1.35 / 0.75 µs

Downstream

Checked pyensembl, varcode, vaxrank, mhctools and sercol for overrides of __reduce__, from_dict, _reconstruct_nested_objects and keyword aliases, and for keyword-only __init__ args. Nothing conflicts. A test-suite run of those five repos against 1.1.1 vs this branch is in progress; I'll post the results here.

./lint.sh and ./test.sh pass (57 tests).

https://claude.ai/code/session_01JRDMg3F5kNKmnjv5jvQBgr

@coveralls

coveralls commented Oct 6, 2026 •

Copy link
Copy Markdown

Coverage Report for CI Build 37522182131

Coverage decreased (-1.0%) to 91.026%

Details

  • Coverage decreased (-1.0%) from the base build.
  • Patch coverage: No coverable lines changed in this PR.
  • 21 coverage regressions across 3 files.

Uncovered Changes

No uncovered changes found.

Coverage Regressions

21 previously-covered lines in 3 files lost coverage.

File Lines Losing Coverage Coverage
helpers.py 12 91.78%
primitive_types.py 6 40.0%
serializable.py 3 93.62%

Coverage Stats

Coverage Status
Relevant Lines: 234
Covered Lines: 213
Line Coverage: 91.03%
Coverage Strength: 0.91 hits per line

💛 - Coveralls

@iskandr
iskandr force-pushed the fix-generic-docstrings branch from ef8cf34 to 9ec4a04 Compare October 6, 2026 16:56
@iskandr
iskandr force-pushed the faster-pickle-and-decode branch 2 times, most recently from 5f0f0f0 to 57cc43f Compare October 6, 2026 19:48
@iskandr
iskandr force-pushed the fix-generic-docstrings branch from e1cf7f3 to 5eb7c8b Compare October 6, 2026 19:51
@iskandr
iskandr changed the base branch from fix-generic-docstrings to master October 6, 2026 19:52
- Pickle through from_dict(to_dict()) instead of converting the whole
  object tree to its JSON-shaped representation first: pickle.dumps is
  ~3x and pickle.loads ~4.5x faster, and pickles are ~35% smaller.
  Values in to_dict() are now pickled natively. Pickles written by
  earlier versions still load via from_serializable_repr.
- from_json reconstructs objects bottom-up in a json object_hook rather
  than walking the parsed tree a second time (~1.6x faster).
- from_serializable_repr no longer pops keys out of its input, so
  decoding the same representation twice works (it used to raise
  IndexError), and Serializable.from_dict no longer renames aliased keys
  in the caller's dict.
- Cache __init__ argument names per class; this runs on every default
  to_dict/__eq__/__hash__. Drops the __new__ fallback, which could never
  succeed on Python 3.
- Document (README, docs/guides/objects.md) and test that keyword-only
  __init__ args are excluded from
  the default to_dict(); varcode's SpliceOutcomeSet depends on it.
  Add a pickling section to the objects guide.

The JSON wire format is unchanged.

Claude-Session: https://claude.ai/code/session_01JRDMg3F5kNKmnjv5jvQBgr
@iskandr
iskandr force-pushed the faster-pickle-and-decode branch from 57cc43f to c1d173b Compare October 6, 2026 19:52
@iskandr
iskandr merged commit 78befc3 into master Oct 6, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants