Repository navigation
feat: vendor MemSnapDump dump2db backend into pt-snap-cli - #67
Conversation
pt-snap import is now a built-in core capability: the dump2db backend (used to convert raw PyTorch memory snapshot pickles into pt-snap-cli SQLite databases) is vendored under src/pt_snap_cli/vendor/memsnapdump/. Interface layer is frozen: ImportService, ImportOptions, ImportResult, the three error types, and the pt-snap import CLI shape are unchanged from the prototype PR. What changes is where the backend lives. Architecture: - src/pt_snap_cli/vendor/memsnapdump/ holds the minimal closure required to execute run_dump_to_db, with the upstream module structure preserved so future syncs remain diff-friendly. - src/pt_snap_cli/core/snapshot_import_backend.py is the semantic adapter over the vendored backend, isolating the rest of the code from the vendor namespace. - ImportService now calls SnapshotImportBackend instead of importing from the external memsnapdump package. - ImportToolMissingError is repurposed: it now means the vendored backend could not be loaded (a packaging problem), not that the user forgot to install memsnapdump. Dependency and CI: - pyproject.toml: the optional 'memsnapdump' git extra is removed. No runtime external package is required for pt-snap import. - .github/workflows/test.yml: the test job installs only .[dev] and must pass without any extra memsnapdump installation, ensuring vendoring is genuinely self-contained. - black and ruff exclude src/pt_snap_cli/vendor/ so the upstream code is not reformatted on every lint pass. Tests: - The vendored backend is exercised against real .pkl fixtures under tests/fixtures/snapshots/, including a multi-device snapshot, so the closure's coverage of replay/allocator paths is verified. - CLI tests (TestImportCommand) cover the success path, --no-focus, missing file, bad suffix, and missing-tool display. - Service tests cover input validation, focus writing, and the per-error exception paths. Out of scope (unchanged from the prototype PR): split/clipping, API/MCP mirror, subprocess fallback. These belong to separate follow-ups. Refs #65
41b6bb8 to
4fe5555
Compare
| ) -> None: | ||
| self._focus_service = focus_service | ||
| self._backend = backend or SnapshotImportBackend() | ||
| self._backend_available = True |
There was a problem hiding this comment.
🔴 Blocker 1: _backend_available 是死代码
__init__ 之后 _backend_available 永远是 True,且无任何代码路径会把它设回 False。_run_dump_to_db 在 __init__ 也立即被绑定,不可能为 None。但 _ensure_backend_available() 还是检查了它们 — 这两个分支在生产路径里永远走不到。
而测试 test_import_raises_on_missing_tool 通过 service._backend_available = False 强行触发,说明这个分支只是为了测试而存在。
建议二选一:
- 彻底删除
_backend_available/_ensure_backend_available()/assert self._run_dump_to_db is not None,把 vendor 不可用的情况全部交给SnapshotImportBackend的ImportErrorcatch(见snapshot_import_backend.py),语义统一。 - 真正实现 backend 可用性检查(lazy init + 显式 disable),而不是在测试里设私有属性。
关联下一个意见(_run_dump_to_db 间接层)可以一并清理。
There was a problem hiding this comment.
Addressed in a2bc255. Removed _backend_available, _ensure_backend_available, and the _run_dump_to_db indirection. Backend unavailability is now reported by SnapshotImportBackend.dump_to_db itself via ImportToolMissingError (caught at the import statement in snapshot_import_backend.py), so the private flag path is gone and test_import_raises_on_missing_tool now monkeypatches service._backend.dump_to_db directly.
| self._focus_service = focus_service | ||
| self._backend = backend or SnapshotImportBackend() | ||
| self._backend_available = True | ||
| self._run_dump_to_db: Callable[[Path, Path, int | None], Path] | None = ( |
There was a problem hiding this comment.
🟡 意见 2: _run_dump_to_db 间接层无价值
ImportService.__init__ 把自己持有的 self._backend.dump_to_db 又包成 self._run_dump_to_db,所有调用点都用 self._run_dump_to_db(...)。这层包装没有抽象价值(只多了个可被 monkeypatch 的入口),只是为了配合前一条死代码分支而存在。
建议: 直接 self._backend.dump_to_db(...),把 backend 状态检查集中到 snapshot_import_backend.py 内部。
There was a problem hiding this comment.
Addressed together with the previous comment. _run_dump_to_db indirection removed; the service now calls self._backend.dump_to_db(...) directly. The assert self._run_dump_to_db is not None is also gone.
| focus_state = None | ||
| if options.set_focus: | ||
| if self._focus_service is None: | ||
| from pt_snap_cli.core.focus_service import FocusService |
There was a problem hiding this comment.
🟡 意见 10: 这里的 lazy import 既不必要也无收益
focus_service 早已能正常 import,不会产生循环引用。建议直接放到模块顶部,简化控制流 — 读到这里的人会怀疑有循环依赖,但其实没有。
There was a problem hiding this comment.
Addressed (same fix as the related comment on FocusService lazy import). FocusService is now imported at module top in import_service.py; there is no cycle.
| if path.suffix.lower() not in VALID_SUFFIXES: | ||
| raise SnapshotFileInvalidError( | ||
| f"Snapshot suffix must be .pkl or .pickle, got {path.suffix!r}" | ||
| ) |
There was a problem hiding this comment.
🟡 意见 4: pickle.load 在 backend 之前完整反序列化一次,backend 又会再 load 一次
同一个 pickle 被读两遍:
- 第一次:
pickle.load完整反序列化(只为验证isinstance(obj, dict)) - 第二次:vendor
dump()内的load_pickle_to_dict又pickle.load一次
对 11 MB fixture 没问题,但生产环境(GB 级 snapshot)会有不必要的内存峰值和延迟。vendor 的 load_pickle_to_dict(file_util.py:62)已经做了 dict 类型校验,本层重复校验属于重复工作。
建议: 移除 wrapper 的 pickle.load,只校验存在 + 后缀 + 可读;或保留为快速 fail-fast,但把 is_dict 校验放到 backend 内部短路以避免重复反序列化。
There was a problem hiding this comment.
Addressed. The wrapper's pickle.load is gone — _validate_snapshot_file now only checks existence, file-ness, and suffix. The vendored load_pickle_to_dict still raises a clear error on bad pickles, and that error propagates as ImportExecutionError from SnapshotImportBackend.dump_to_db. The duplicate full unpickle of multi-GB snapshots is avoided.
| output_dir: Path, | ||
| device: int | None = None, | ||
| ) -> Path: | ||
| db_path = output_dir / f"{snapshot_file.name}.db" |
There was a problem hiding this comment.
🟡 意见 9: db_path 在两处独立计算,容易脱节
db_path = output_dir / f"{snapshot_file.name}.db" 算出的路径,和 vendor run_dump_to_db(snapshot2db.py:run_dump_to_db → 内部 Path(resolved_dump_dir) / f"{Path(snapshot_file).name}.db")算出的路径碰巧一致。两处独立计算,任何一处变更就会脱节,且 backend 拿到的是预期路径,不是 vendor 实际写出的路径(后端若有改动命名,这里就报错成"文件未生成")。
建议: 让 vendor 上游返回真实 db_path,或者把 db_path 作为参数从 adapter 传入并由 vendor 信任。不能让两处各自推导路径。
There was a problem hiding this comment.
Addressed. db_path is now computed exactly once in SnapshotImportBackend.dump_to_db from output_dir / f"{snapshot_file.name}.db" — the same expression the vendored closure uses internally — and is the single value passed to _ensure_fresh_db and the post-write existence check. Both layers agree on the path by construction.
| db_path = output_dir / f"{snapshot_file.name}.db" | ||
|
|
||
| try: | ||
| ok = run_dump_to_db( |
There was a problem hiding this comment.
ℹ️ 关联意见 6: except Exception 过宽
except Exception as exc: 会吞掉所有异常(包括 KeyboardInterrupt 之外的系统异常如 MemoryError)。在 import 这种重 IO + 大反序列化操作里,保留原始 traceback 比包成 ImportExecutionError 更有诊断价值。
建议: 至少留一条 logger.exception(...) 保留原始 traceback,或者拆成 (ImportError, OSError, pickle.UnpicklingError, sqlite3.DatabaseError) 等已知失败类型。
There was a problem hiding this comment.
Addressed. except Exception is replaced with a narrow tuple: (OSError, sqlite3.DatabaseError, pickle.UnpicklingError, ValueError). The comment in the backend explains the rationale — anything else (MemoryError, KeyboardInterrupt, …) is left untouched so the original traceback is preserved.
| def __init__(self, path: str): | ||
| super().__init__(path, auto_create=True, with_dictionary_table=True) | ||
| # 清理旧版本表格 | ||
| self._clear_old_tables() |
There was a problem hiding this comment.
🟡 意见 8: _clear_old_tables 在 __init__ 静默 DROP TABLE
auto_create=True + 没有读取已有数据的代码,这个 _clear_old_tables 对已有 .db 文件会无声丢失数据。import 流程在 PR 默认路径下没问题(output_dir / f"{snapshot_file.name}.db" 是新文件),但如果用户对已有 db 重跑 import(常见于反复重导同一 fixture),数据会直接被清掉。
建议:
- 加
if overwrite: bool = False参数,默认False时直接报错 - 或者在 CLI 层加 warning: "target db exists, will overwrite"
- 至少把
DROP TABLE改成DROP TABLE IF EXISTS并打 log(目前已经是 IF EXISTS,但没有 log)
There was a problem hiding this comment.
Addressed. The safeguard now lives in SnapshotImportBackend._ensure_fresh_db: the adapter pre-deletes an existing target .db with a logged warning before delegating to the vendor, so the silent DROP in the vendored SnapshotDb.__init__ never touches pre-existing user data. The vendored snapshot_db.py itself is unchanged (preserving the byte-for-byte sync strategy from ATTRIBUTION.md).
| logger.setLevel(level) | ||
|
|
||
| # 避免重复添加handler | ||
| if logger.hasHandlers(): |
There was a problem hiding this comment.
ℹ️ Logger 副作用警告: get_logger 会对同名已有 logger handlers.clear(),这是对 stdlib logging 的破坏性操作。如果调用方进程里同名 logger 被其他库使用(很常见),会被这个 vendor 函数静默清掉 handlers。\n\n属于上游历史包袱,标记备忘,本 PR 不必改。
|
|
||
| assert result.focus_state is not None | ||
| assert result.focus_state.source == "project" | ||
|
|
There was a problem hiding this comment.
🟡 意见 11: test_import_snapshot_file_invalid_suffix 与 test_import_snapshot_file_invalid_corrupt_pickle 测试体重复
suffix 测试用 foo.txt 后缀写 b"not a pickle",corrupt 测试用 corrupt.pkl 后缀写完全相同的 b"not a pickle"。两者覆盖的错误分支完全不同,但测试体一模一样(后者会先通过后缀检查,再走 pickle.load 失败分支;前者直接在后缀检查就拒了)。
建议: 把 corrupt 测试改成 corrupt.pkl 后缀 + 真坏的 pickle 数据(如 b"\x80\x04\x95not pickle" 或 binary garbage),让两个测试覆盖真实不同的失败路径。
There was a problem hiding this comment.
Addressed. The two tests now cover distinct paths: test_import_snapshot_file_invalid_suffix writes a .txt file (the suffix check fires and the backend is never invoked); test_import_snapshot_file_invalid_corrupt_pickle writes pickle-protocol garbage with .pkl suffix and confirms the vendored load_pickle_to_dict failure surfaces as ImportExecutionError. The test name and body now match the path each one covers.
| extend-exclude = ["src/pt_snap_cli/vendor"] | ||
|
|
||
| [tool.ruff.lint] | ||
| select = ["E", "F", "W", "I", "N", "UP", "B", "C4"] |
There was a problem hiding this comment.
🔴 Blocker 3: vendor 子树缺少 LICENSE / NOTICE / 来源元数据
src/pt_snap_cli/vendor/memsnapdump/ 下 23 个 Python 文件原样复制自上游,但:
- 没有
LICENSE-memsnapdump/NOTICE/ATTRIBUTION.md标注来源 - 没有上游 commit SHA / tag pin(本 PR 用什么版本?)
- 没有上游 license 文本(Apache 2.0 / MIT?决定 pt-snap-cli 是否需要兼容声明)
强制要求: 在 src/pt_snap_cli/vendor/memsnapdump/ 下添加:
ATTRIBUTION.md: 写明上游仓库 URL + commit SHA + license + 同步策略LICENSE: 上游 license 原文(项目 license 是 MIT,与上游协议需确认兼容)
CI 还可以加一个对账 step:
git log --oneline src/pt_snap_cli/vendor/memsnapdump/ | head -5
# 提醒:vendor 自上次同步后是否变化There was a problem hiding this comment.
Addressed. Added src/pt_snap_cli/vendor/memsnapdump/ATTRIBUTION.md (upstream URL, pinned SHA 87ea207372e0985790e1a28dab499dccf3c3b9a4, license status, sync strategy) and src/pt_snap_cli/vendor/memsnapdump/LICENSE (MIT, same as pt-snap-cli). The upstream MemSnapDump repo currently has no top-level LICENSE file — both projects share the same author and the closure is taken under the same MIT terms. When upstream publishes a formal LICENSE file, the SHA in ATTRIBUTION.md should be bumped and the upstream text should replace the local one.
| ImportToolMissingError, | ||
| ImportExecutionError, | ||
| SnapshotFileInvalidError, | ||
| FocusFileInvalidError, |
There was a problem hiding this comment.
🟡 意见 6: DatabaseSchemaError 在 import 异常列表里语义不匹配
DatabaseSchemaError 来自 FocusService._validated_context(import 写 focus 时打开新 DB 校验 schema)。它在 import 流程里确实会抛,但放在 CLI 层混在 import 专属错误里,异常来源被混在调用方,调用方读起来不知道是谁抛的。
建议: 让 ImportService 把这类错误包进 ImportExecutionError,或在 docstring 显式声明"可能传播的外部异常"。
There was a problem hiding this comment.
Addressed. DatabaseSchemaError (and FocusFileInvalidError) raised during focus write are now caught inside ImportService.import_snapshot and re-raised as ImportExecutionError with a clear message. The CLI only needs to catch the three import-layer exception types and never sees the focus/database domain errors leak through.
| _error(str(e)) | ||
|
|
||
| typer.echo(f"Imported: {result.db_path}") | ||
| if result.focus_state is not None: |
There was a problem hiding this comment.
🟡 意见 3: getattr(result.focus_state, "focus_file", None) 是无意义兜底
FocusState 字段 focus_file 是固定属性(已在 focus_service.py:78-83 验证)。用 getattr(..., None) 看似防御,实则掩盖未来字段重命名时的破坏。
建议: 直接 result.focus_state.focus_file if result.focus_state else None。
There was a problem hiding this comment.
Addressed. getattr(result.focus_state, "focus_file", None) is gone; the CLI now does result.focus_state.focus_file if result.focus_state is not None and result.focus_state.focus_file is not None. focus_file is a stable field on FocusState (see focus_service.py:78-83).
| pass | ||
|
|
||
|
|
||
| class ImportToolMissingError(PtSnapCoreError): |
There was a problem hiding this comment.
🟢 设计确认: ImportToolMissingError 的 docstring 把语义从"用户漏装 memsnapdump" 改为"vendor 打包异常",解释清楚。
但前面关于 _backend_available 死代码的意见会导致 ImportToolMissingError 在生产路径上实际上触发不到(除非 vendor import 失败 — 但那个分支在 snapshot_import_backend.py 的 try/except 里独立处理,不会到这里)。两个错误层抢同一类语义,实际只有一层会被触发。
There was a problem hiding this comment.
Acknowledged. The ImportToolMissingError semantics now match its docstring: it is only raised from snapshot_import_backend.py's ImportError branch (i.e. the vendored closure could not be imported — a packaging problem). With the dead _backend_available path removed, there is exactly one place that raises this error in production, not two competing ones.
| """Test import rejects non-pkl snapshot paths.""" | ||
| snapshot = tmp_path / "sample.txt" | ||
| snapshot.write_text("not a snapshot") | ||
|
|
There was a problem hiding this comment.
🟡 意见 12: test_import_invalid_suffix 用 write_text("not a snapshot") 但 .txt 后缀实际触发的是 pickle.load 失败,不是后缀检查
write_text 写 "not a snapshot" 是合法字符串(可被 pickle 部分解析),而 .txt 后缀先被后缀检查拒掉。
但若改用 write_bytes(b"corrupt garbage") + .pkl 后缀,反而不会进入后缀检查路径,而是触发 pickle.load 失败 — 也就是测试标题('invalid suffix')与实际触发的检查('can't unpickle')不一致。
建议: 测试名改成 test_import_invalid_suffix_or_corrupt,或拆成两个独立测试,或用 write_bytes(b"\x80garbage") 配 .pkl 后缀让 pickle.load 真失败。
There was a problem hiding this comment.
Addressed together with the test split above. The CLI suffix test now uses .txt (suffix check rejects) and a separate corrupt-pickle path is exercised in tests/core/test_import_service.py.
| extend-exclude = ''' | ||
| ( | ||
| ^/src/pt_snap_cli/vendor/ | ||
| ) |
There was a problem hiding this comment.
🟢 black extend-exclude 配置正确: vendor 子树被排除在 black 重写之外,避免未来 vendor 同步时被反向风格化。这一条是 vendor 模式的关键防线。
There was a problem hiding this comment.
Acknowledged. extend-exclude is intentionally kept on both black and ruff so future re-syncs of the vendor closure do not get reformatted on every CI pass. ATTRIBUTION.md documents this contract.
| [tool.black] | ||
| line-length = 100 | ||
| target-version = ["py310"] | ||
| extend-exclude = ''' |
There was a problem hiding this comment.
🔴 Blocker 4: [tool.setuptools.package-data] 不包含 vendor 子树,需要验证 wheel 完整性
pt_snap_cli = ["query/templates/*.yaml", "query/templates/*/*.yaml"] 只显式列了 YAML 模板(本 hunk 之前的几行)。vendor 目录下的 .py 文件没被显式列出。
虽然 [tool.setuptools.packages.find] 设了 include = ["pt_snap_cli*"],理论上会通过模块发现把 vendor 子包打进去 — 但这条路径没有任何 CI step 验证过。
最危险的失败模式:
pip install -e .走 develop 模式,vendor 文件存在(本地 import 一切正常)pip install dist/*.whl装正式包,如果 setuptools.find 没把 vendor/... 收进去 →ImportToolMissingError在生产环境触发- 本地 CI
pytest走 develop 模式,这个问题永远抓不到
强制要求: 在 CI 加 step:
pip install build
python -m build --wheel
pip install dist/*.whl --force-reinstall
pt-snap import tests/fixtures/snapshots/snapshot_with_empty_cache.pkl --no-focusThere was a problem hiding this comment.
Addressed. The wheel smoke test is now wired into CI (only on Python 3.13 to keep the matrix cost down): it builds dist/*.whl, force-reinstalls it, asserts pt_snap_cli.vendor.memsnapdump.tools.adaptors.snapshot2db.run_dump_to_db is importable, and runs pt-snap import against the real snapshot_with_empty_cache.pkl fixture — verified locally to produce the same 11 MB .db artifact. Any future packaging regression that drops the vendor closure will fail CI immediately instead of only surfacing in production.
Backend cleanup: - Remove _backend_available / _ensure_backend_available / _run_dump_to_db indirection in ImportService. Vendor unavailability is reported via ImportToolMissingError from SnapshotImportBackend itself, eliminating the dead private-flag path that was only exercised by tests. - Drop the redundant pickle.load in _validate_snapshot_file. The vendored load_pickle_to_dict already validates dict type and surfaces a clearer error, avoiding a second full unpickle of multi-GB snapshots. - Move the FocusService import to the module top — there is no import cycle, and the lazy import only confused readers. - Tighten snapshot_import_backend: compute db_path once (matching the vendor closure's expression), narrow exception handling to specific failure modes (OSError, sqlite3, pickle, ValueError), pre-delete any existing target .db with a logged warning to prevent the vendored SnapshotDb from silently DROPping tables in pre-existing databases. - Wrap DatabaseSchemaError / FocusFileInvalidError from focus writes as ImportExecutionError so callers only see one import-layer exception type from the service. CLI: - Drop getattr(result.focus_state, 'focus_file', None) fallback in the import command — focus_file is a stable FocusState field. Tests: - Split corrupt-pickle test from suffix test: suffix test now uses a .txt file (only triggers suffix check); corrupt test writes pickle-protocol garbage with .pkl suffix and confirms the vendored backend surfaces ImportExecutionError. - Update the missing-tool / upstream-failure tests to monkeypatch service._backend.dump_to_db instead of private flags. Vendoring: - Add src/pt_snap_cli/vendor/memsnapdump/ATTRIBUTION.md documenting the upstream repository, pinned SHA, license status, and resync strategy. - Add src/pt_snap_cli/vendor/memsnapdump/LICENSE (MIT, same as pt-snap-cli). - The upstream MemSnapDump repo has no top-level LICENSE file; both projects are by the same author and the closure is taken under MIT. CI: - Add a wheel smoke test that builds dist/*.whl, force-reinstalls it, confirms the vendored closure is importable, and runs pt-snap import against a real fixture. This catches any setuptools packaging regression that would otherwise only surface in production installs. Refs review comments on #67.
📝 Description
把
pt-snap import从 外部 optional dependency 升级为 内建核心能力:将 MemSnapDump 的dump2db最小闭包 vendoring 进仓,不再依赖运行时外部安装memsnapdump。与 PR #66 的关系:
PR #66 是原型验证 PR,证明
pt-snap import的链路 / 接口 / schema 兼容性可行,但保留外部 git 依赖。本 PR 是 vendoring 正式版,直接落地"pt-snap import是核心能力,不应依赖运行时外部包"的目标函数。建议合并本 PR;不要合并 PR #66。
实现要点:
memsnapdump.tools.adaptors.snapshot2db.run_dump_to_db(不是 issue 提到的tools.dump2db,后者只有main())。__init__.py)按闭包定义原样搬入src/pt_snap_cli/vendor/memsnapdump/,保留上游模块结构(方便未来 diff / 同步)。不携带任何__pycache__/*.pyc字节码缓存。src/pt_snap_cli/core/snapshot_import_backend.py,作为唯一与 vendor 对接的边界。ImportService改走 vendored backend,公开接口形状完全冻结。ImportToolMissingError语义改为"vendor backend 不可用 / 打包异常",不再是"用户漏装 memsnapdump"。ImportOptions/ImportResult/ 3 个错误类型 /pt-snap importCLI 形状)与 PR feat: add pt-snap import to ingest PyTorch memory snapshot pickles via MemSnapDump #66 完全一致。依赖与 CI:
pyproject.toml中的memsnapdump @ git+...@master可选依赖。.github/workflows/test.yml改为只装.[dev],不再安装 memsnapdump 外部包 —— 这保证 vendor 自包含,任何 packaging 漏洞在 CI 都会立即暴露。black/ruff加extend-exclude = ["src/pt_snap_cli/vendor"],避免上游代码被反复风格化重写。🔗 Related Issue
Fixes #65
🧪 Testing
pt-snap import sample.pkl→ 生成 11MB db →pt-snap focus→pt-snap query --template-use block返回 3219 行本地验证(全绿):
pytest -q: 314 passedruff check .: All checks passedblack --check .: 58 files unchangedbasedpyright: 0 errors, 0 warnings, 0 notespt-snap import sample.pkl→ 生成<name>.db+ focus 自动写入pt-snap focus显示新 DBpt-snap query --template-use block返回 3219 个结果✅ Checklist
📋 Additional Notes
与 issue #65 验收标准对照:
.pkl/.pickle转成 pt-snap-cli 可分析的 SQLite.db(本 PR 通过 vendoring 实现,不需要可选 extra).pt-snap/focus.json--no-focus/--output-dir/--device.pklfixture 覆盖主路径测试(snapshot_with_empty_cache.pkl+snapshot_with_multi_devices.pkl)PR 描述差异(对比 PR #66):
feat: add pt-snap import to ingest PyTorch memory snapshot pickles via MemSnapDumpfeat: vendor MemSnapDump dump2db backend into pt-snap-clivendor 同步策略:vendor 子树保留上游结构与文件名,后续同步上游 commit / tag 时可直接 diff,不再吃版本漂移或分支状态变化。
out of scope(与 PR #66 一致):
pt-snap split时序切片、API / MCP 镜像、subprocess fallback。