Skip to content

refactor serialize_json to use custom JSONEncoder for ~3.7x speedup#450

Merged
essweine merged 1 commit intofeature/performancefrom
feature/performance2
Apr 17, 2026
Merged

refactor serialize_json to use custom JSONEncoder for ~3.7x speedup#450
essweine merged 1 commit intofeature/performancefrom
feature/performance2

Conversation

@danfunk
Copy link
Copy Markdown
Collaborator

@danfunk danfunk commented Apr 16, 2026

Instead of two passes (Python dict-building + json.dumps), serialize_json now uses a SpiffEncoder that lets the C-level JSON encoder handle plain data traversal natively, only calling back to Python for registered types.

Periodic serialization test (300 items, 180 serializations):

Metric Before After Improvement
Total time 162.5s 45.0s 3.6x faster
Total serialization 160.7s 43.3s 3.7x faster
Avg per serialization 0.893s 0.241s 3.7x faster
Final checkpoint (308 tasks) 2.07s 0.56s 3.7x faster

Instead of two passes (Python dict-building + json.dumps), serialize_json
now uses a SpiffEncoder that lets the C-level JSON encoder handle plain
data traversal natively, only calling back to Python for registered types.

Periodic serialization test (300 items, 180 serializations):

| Metric                         | Before   | After   | Improvement   |
|--------------------------------|----------|---------|---------------|
| Total time                     | 162.5s   | 45.0s   | 3.6x faster   |
| Total serialization            | 160.7s   | 43.3s   | 3.7x faster   |
| Avg per serialization          | 0.893s   | 0.241s  | 3.7x faster   |
| Final checkpoint (308 tasks)   | 2.07s    | 0.56s   | 3.7x faster   |

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@essweine
Copy link
Copy Markdown
Contributor

I guess this is not terrible, though all these changes are not really addressing the problem, which is that if you need to work with large amounts of data attached to each task, the library isn't the appropriate place to deal with it.

@essweine essweine merged commit 68ef142 into feature/performance Apr 17, 2026
5 checks passed
@essweine essweine deleted the feature/performance2 branch April 17, 2026 15:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants