Skip to content

feat: make snapshot runtime first-party and add replay-safe splitting - #81

Merged
codeasier merged 12 commits into
mainfrom
memsnapdump-first-party
Aug 1, 2026
Merged

codeasier merged 12 commits into
mainfrom
memsnapdump-first-party

Conversation

@codeasier

@codeasier codeasier commented Jul 31, 2026 •

Copy link
Copy Markdown
Owner

Description

  • Migrate the retained snapshot runtime from pt_snap_cli.vendor.memsnapdump to the first-party pt_snap_cli.snapshot namespace.
  • Add shared pickle/JSON representation loading, canonicalization, replay validation, and provenance/package metadata.
  • Add replay-safe pt-snap split support with per-device slicing, deterministic output names, same-filesystem staging, validation, and conflict-safe atomic publication.
  • Add bilingual documentation, trusted-pickle warnings, provenance governance, migration audits, and executable verification runbooks.
  • Establish the verified post-migration performance baseline for follow-up issue [FEATURE] 优化 import 路径数据结构,消除 segment/blocks 操作的 O(N²) 退化(子任务 #72-2) #74 work.

Related Issue

Related to #74. This PR establishes the first-party runtime and verified baseline that issue #74 should rebase onto.

Testing

  • Tests added/updated
  • Manual testing completed

Verified against rewritten PR head 32fb646c6bcf8959a5882e89a6965d13bec37513: 565 passed, 1 expected xfail; Ruff, Black, BasedPyright, provenance, Python 3.10-3.13, and wheel smoke passed. Import correctness and wall-clock gates passed. The 628k Max RSS measurements remained variable across controlled reruns and require separate performance review; internal benchmark handoff artifacts are intentionally excluded from this PR.

Checklist

  • Code follows project guidelines
  • Documentation updated
  • No new warnings introduced
  • Changes tested locally

Snapshot Runtime Provenance

  • Snapshot provenance evaluation completed when snapshot runtime files change
    Snapshot provenance decision: updated
    Snapshot provenance no-update reason: not applicable

Additional Notes

  • Pinned upstream source: 87ea207372e0985790e1a28dab499dccf3c3b9a4.
  • Source provenance is tracked in src/pt_snap_cli/snapshot/PROVENANCE.md; relicensing is recorded in docs/legal/memsnapdump-mit-relicensing.md.
  • The original private benchmark fixtures remain ignored and untracked. Their sanitized 131k and 628k replacements are stored with Git LFS and are used by benchmarks/baseline_import.py.
  • Review fixes are retained in the rewritten history; no runtime changes follow d285ef3.

@codeasier codeasier left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code review of the split engine and snapshot runtime migration. Test suite verified: 559 passed, 1 xfailed at the PR head; the migration is behavior-preserving and the split/replay logic is well-tested. Three findings below, with severity and reproduction paths.

Comment thread src/pt_snap_cli/core/errors.py Resolved
Comment thread src/pt_snap_cli/core/__init__.py Resolved
Comment thread src/pt_snap_cli/core/models.py Resolved
Comment thread src/pt_snap_cli/core/split_service.py Resolved
Comment thread src/pt_snap_cli/core/split_service.py Resolved

@codeasier codeasier left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the first-party snapshot runtime migration and replay-safe split. No blocking issues found; tests/lint pass. Two minor nits below on the new hooker.py.

Comment thread src/pt_snap_cli/core/split_service.py Resolved

@codeasier codeasier left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review summary - 2 inline findings (1 medium, 1 low):

  1. hooker.py:95-111 - state mutated before write (medium): If the file write raises, the hooker is left with prev_segments deleted and events_buffer cleared, causing a masking AttributeError on the next dump and silent event loss. See inline comment for the suggested fix.

  2. entities.py:261 - off-by-one (low, pre-existing): <= should be <. Not reachable via the new CLI path but a trivial one-word fix.

Additional observation (not blocking): split_service.py:196-221 - the _stage_identity / _verify_stage_identity pair has a narrow theoretical symlink-substitution race between lstat() and os.open(O_DIRECTORY) on POSIX. Already mitigated by mkdtemp in the user's own output dir + fd retention; a documenting comment about the assumption would be reasonable but no code change needed.

What looks correct: SplitError.__reduce__ (pickle-safe), mcp<2 pin, append+reverse trace ordering, staging-fd inode-reuse fix, provenance CI guard, and trace-id round-trip all verified with test coverage.

@codeasier
codeasier force-pushed the memsnapdump-first-party branch from ca34de8 to 69e26a6 Compare August 1, 2026 02:58
@codeasier
codeasier merged commit 717e3ca into main Aug 1, 2026
12 checks passed
@codeasier
codeasier deleted the memsnapdump-first-party branch August 1, 2026 05:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant