Skip to content

feat: add pt-snap import to ingest PyTorch memory snapshot pickles via MemSnapDump - #66

Closed
codeasier wants to merge 3 commits into
mainfrom
feat/memsnapdump-integration
Closed

codeasier wants to merge 3 commits into
mainfrom
feat/memsnapdump-integration

Conversation

@codeasier

Copy link
Copy Markdown
Owner

📝 Description

集成 codeasier/MemSnapDump 的 dump2db 能力,新增 pt-snap import 子命令,将原始 PyTorch memory snapshot(.pkl / .pickle)一键转换为 pt-snap-cli 已支持分析的 SQLite .db,并自动写入项目级 focus,使后续 pt-snap query / pt-snap report 无需重复指定路径。

实现要点:

  • 真实入口为 memsnapdump.tools.adaptors.snapshot2db.run_dump_to_db(issue 提到的 memsnapdump.tools.dump2db 仅暴露 main(),无法直接调用)。
  • memsnapdump 声明为 optional extra(memsnapdump @ git+https://github.com/codeasier/MemSnapDump.git@main),源码集成,缺包时给出清晰安装提示,不做 subprocess fallback。
  • schema 完全兼容:MemSnapDump 输出列名(camelCase:requestedSize / allocEventId / freeEventId)与现有 query 模板一致,无需新增 schema / 模板 / 修改 Context 核心逻辑(仅补一个 discover_devices() 便利别名)。
  • ImportService 采用 lazy import + 单一私有调用点 _run_dump_to_db(便于 monkeypatch 测试)。

🔗 Related Issue

Fixes #65

🧪 Testing

  • Tests added/updated — 22 个新测试(models / errors / ImportService 含真实 fixture 主路径 / CLI 命令)
  • Manual testing completed — 端到端:pt-snap import snapshot.pkl → 生成 11MB db → pt-snap focus 显示新 db → pt-snap query --template-use block 返回 3219 行

测试覆盖:

  • tests/core/test_import_models.py(4):ImportOptions/ImportResult 字段与冻结性
  • tests/core/test_import_errors.py(3):3 个错误类型继承 PtSnapCoreError
  • tests/core/test_import_service.py(10):主路径(真实 fixture → db → focus)、多设备、5 类错误条件、--no-focus、.pickle 后缀输出命名
  • tests/test_cli.py::TestImportCommand(5):CLI 成功 / 缺包提示 / 非法路径 / 非法后缀 / --no-focus

Fixture:从 MemSnapDump tests/test_data/ 引入 2 个真实 .pkl(603K + 1.3M,合计 1.9M)到 tests/fixtures/snapshots/,作为集成测试主路径。

验证结果:

  • pytest:314 passed(292 基线 + 22 新增,0 失败)
  • ruff check .:All checks passed
  • black --check .:57 files unchanged
  • basedpyright:0 errors(仅 memsnapdump 无类型标注导致的 reportAny 警告)

✅ Checklist

  • Code follows project guidelines
  • Documentation updated (README / docs/en|zh/quickstart / docs/en/database)
  • No new warnings introduced(ruff / black / basedpyright 全绿)
  • Changes tested locally

📋 Additional Notes

与 issue #65 验收标准对照:

  • pyproject.toml 新增 memsnapdump optional extra,不加入默认 runtime dependencies
  • 新增 pt-snap import SNAPSHOT_FILE,支持 --output-dir / --device / --no-focus
  • .pkl / .pickle 输入生成 <snapshot_file.name>.db
  • 默认写入项目级 .pt-snap/focus.json,后续 pt-snap query / pt-snap report 可复用
  • 缺少 memsnapdump 时返回清晰错误与安装提示
  • 使用 MemSnapDump 真实 .pkl fixture 覆盖主路径测试
  • 不实现 pt-snap split / clipping,不新增 API/MCP 镜像,不新增 subprocess fallback
  • 现有测试全部通过,并新增 core service + CLI 覆盖

入口路径纠正:issue 参考资源里写的 memsnapdump.tools.dump2db.run_dump_to_db 实际只有 main(),真正可调用的是 memsnapdump.tools.adaptors.snapshot2db.run_dump_to_db,本 PR 已用正确路径。

安装方式:MemSnapDump 不在 PyPI、无 git tag,故 optional extra 用 git+https://...@main 锁定最新主分支;本地开发可用 pip install -e /path/to/MemSnapDump editable 安装。

后续 PR(out of scope):pt-snap split 时序切片、API/MCP 镜像、subprocess fallback。

codeasier added 3 commits June 15, 2026 06:53
…a MemSnapDump

Integrate codeasier/MemSnapDump's dump2db capability so users can convert a raw
PyTorch memory snapshot (.pkl/.pickle) into a pt-snap-cli analyzable SQLite
database without leaving the CLI. The generated .db is auto-persisted as
project focus, so follow-up query/report commands work without repeating the
path.

Entry point (source-integrated, not PyPI): run_dump_to_db from
memsnapdump.tools.adaptors.snapshot2db — note the actual module path differs
from the issue's reference (memsnapdump.tools.dump2db), which only exposes
main(). memsnapdump is declared as an optional extra pinned to the upstream
main branch via git+https, since the project has no PyPI release or tags.

New surface:
- pt-snap import SNAPSHOT_FILE [--output-dir/-o] [--device/-d] [--no-focus]
- core.ImportService with lazy import + single private call point
  (_run_dump_to_db) for testability; no subprocess fallback
- ImportOptions / ImportResult frozen dataclasses
- ImportToolMissingError / SnapshotFileInvalidError / ImportExecutionError

Schema is fully compatible (MemSnapDump emits camelCase column names
requestedSize/allocEventId/freeEventId which the existing query templates
already use), so no Context / template / schema changes beyond a small
discover_devices() convenience alias.

Tests: 22 new (models, errors, ImportService incl. real-fixture main path,
CLI command). Real .pkl fixtures from MemSnapDump's test_data are committed
under tests/fixtures/snapshots/. 314 total passing, ruff/black/basedpyright
clean.

Out of scope per issue #65: split/clipping, API/MCP mirror, subprocess
fallback.

Closes #65
… raise

The test workflow only installed .[dev], so the 8 tests exercising the
real dump2db path failed with ImportToolMissingError in CI. Install
.[dev,memsnapdump] so the real-fixture main-path tests run.

basedpyright treats warnings as failures in this project. The import
command had a redundant 'raise typer.Exit(code=1) from None' after
_error(str(e)) — _error is NoReturn, making the raise unreachable.
Drop it to match the existing focus/query command pattern.
The upstream codeasier/MemSnapDump repo uses 'master' as its default
branch, not 'main'. The @main ref failed with 'pathspec main did not
match' during pip install in CI.
focus_state = self._focus_service.set_project_focus(
db_path=db_path,
device_id=options.device,
base_dir=output_dir,

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这里把 base_dir 设为 output_dir,会导致 focus 写到输出 DB 所在目录,而不是用户当前项目目录。比如在项目目录执行 pt-snap import /tmp/snapshot.pkl 时,后续从项目目录运行 pt-snap query 可能找不到刚写入的 focus,这和 PR 描述里“后续无需重复指定路径”的目标不一致。建议让 CLI 显式传入当前工作目录作为 project base,或者调整 set_project_focus 的默认语义。

Comment thread pyproject.toml
"basedpyright>=1.39.3",
]
memsnapdump = [
"memsnapdump @ git+https://github.com/codeasier/MemSnapDump.git@master",

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这里用的是 @master,但 PR 描述和文档里写的是 @main。如果上游默认分支已经是 main,这个 optional dependency 会直接安装失败。建议统一 pyproject.toml、文档和 PR 描述中的分支名。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[FEATURE] 集成 MemSnapDump 的原始快照转 DB 能力

1 participant