Skip to content

perf: tune SQLite snapshot imports - #78

Merged
codeasier merged 2 commits into
mainfrom
issue-75-sqlite-io-tuning
Jul 28, 2026
Merged

codeasier merged 2 commits into
mainfrom
issue-75-sqlite-io-tuning

Conversation

@codeasier

Copy link
Copy Markdown
Owner

Summary

  • Apply journal_mode=MEMORY, synchronous=OFF, and a 64 MiB cache only to SnapshotDb import connections
  • Document realistic large-pickle memory requirements and clarify that --device does not reduce the initial load peak
  • Add PRAGMA isolation, failed-output cleanup, and write-throughput benchmark coverage

Streaming assessment

Standard pickle.Unpickler cannot incrementally yield nested device_traces. Reliable streaming would require a producer format change (for example, per-device shards) or a pickle VM implementation that preserves memo/shared-reference semantics, so it is not included as a short-term change.

Verification

  • PYTHONDONTWRITEBYTECODE=1 PYTHONPATH=src pytest - 323 passed
  • ruff check . - All checks passed
  • PYTHONPATH=src python benchmarks/sqlite_write_throughput.py --rows 100000 --batch-size 1000 - 2.68x write-side speedup on APFS

Closes #75

@codeasier codeasier left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

行内检视聚焦于两个可复现的健壮性与可移植性问题。

Comment thread benchmarks/sqlite_write_throughput.py Resolved
Comment thread tests/test_snapshot_db.py Resolved
@codeasier
codeasier force-pushed the issue-75-sqlite-io-tuning branch from efd0013 to 5fa9309 Compare July 28, 2026 07:06
@codeasier
codeasier merged commit a14ff77 into main Jul 28, 2026
5 checks passed
@codeasier
codeasier deleted the issue-75-sqlite-io-tuning branch October 3, 2026 19:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[FEATURE] 调优 SQLite 写入 PRAGMA 并评估 Pickle 流式加载(子任务 #72-3)

1 participant