Repository navigation
feat: vendor MemSnapDump dump2db backend into pt-snap-cli #67
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -67,10 +67,16 @@ pt_snap_cli = [ | |
| [tool.black] | ||
| line-length = 100 | ||
| target-version = ["py310"] | ||
| extend-exclude = ''' | ||
| ( | ||
| ^/src/pt_snap_cli/vendor/ | ||
| ) | ||
|
Owner
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🟢 black extend-exclude 配置正确: vendor 子树被排除在 black 重写之外,避免未来 vendor 同步时被反向风格化。这一条是 vendor 模式的关键防线。
Owner
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Acknowledged. |
||
| ''' | ||
|
|
||
| [tool.ruff] | ||
| line-length = 100 | ||
| target-version = "py310" | ||
| extend-exclude = ["src/pt_snap_cli/vendor"] | ||
|
|
||
| [tool.ruff.lint] | ||
| select = ["E", "F", "W", "I", "N", "UP", "B", "C4"] | ||
|
Owner
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🔴 Blocker 3: vendor 子树缺少 LICENSE / NOTICE / 来源元数据
强制要求: 在
CI 还可以加一个对账 step: git log --oneline src/pt_snap_cli/vendor/memsnapdump/ | head -5
# 提醒:vendor 自上次同步后是否变化
Owner
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Addressed. Added |
||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -36,3 +36,36 @@ class TemplateRenderError(PtSnapCoreError): | |
|
|
||
| class QueryExecutionError(PtSnapCoreError): | ||
| pass | ||
|
|
||
|
|
||
| class ImportToolMissingError(PtSnapCoreError): | ||
|
Owner
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🟢 设计确认: 但前面关于
Owner
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Acknowledged. The |
||
| """Raised when the vendored snapshot import backend is unavailable. | ||
|
|
||
| In the vendored model, this no longer means "the user forgot to install | ||
| memsnapdump" — the dump2db backend is built in. It means the vendored | ||
| backend could not be imported or initialized, which is a packaging | ||
| or installation problem. | ||
| """ | ||
|
|
||
| pass | ||
|
|
||
|
|
||
| class SnapshotFileInvalidError(PtSnapCoreError): | ||
| """Raised when a candidate snapshot pickle file cannot be used as input. | ||
|
|
||
| Triggers include: path does not exist, path is not a regular file, | ||
| suffix is not .pkl or .pickle, file cannot be unpickled, or the | ||
| unpickled top-level object is not a dict. | ||
| """ | ||
|
|
||
| pass | ||
|
|
||
|
|
||
| class ImportExecutionError(PtSnapCoreError): | ||
| """Raised when the import backend itself fails to produce a database. | ||
|
|
||
| Covers upstream dump2db returning False, raising an exception, | ||
| or producing no .db artifact where one was expected. | ||
| """ | ||
|
|
||
| pass | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,70 @@ | ||
| from __future__ import annotations | ||
|
|
||
| import logging | ||
| from pathlib import Path | ||
|
|
||
| from pt_snap_cli.core.errors import ( | ||
| DatabaseSchemaError, | ||
| FocusFileInvalidError, | ||
| ImportExecutionError, | ||
| SnapshotFileInvalidError, | ||
| ) | ||
| from pt_snap_cli.core.focus_service import FocusService | ||
| from pt_snap_cli.core.models import ImportOptions, ImportResult | ||
| from pt_snap_cli.core.snapshot_import_backend import SnapshotImportBackend | ||
|
|
||
| logger = logging.getLogger(__name__) | ||
|
|
||
| VALID_SUFFIXES = {".pkl", ".pickle"} | ||
|
|
||
|
|
||
| class ImportService: | ||
| """Import PyTorch snapshot pickle files through the vendored backend.""" | ||
|
|
||
| def __init__( | ||
| self, | ||
| focus_service: FocusService | None = None, | ||
| backend: SnapshotImportBackend | None = None, | ||
| ) -> None: | ||
| self._focus_service = focus_service | ||
| self._backend = backend or SnapshotImportBackend() | ||
|
|
||
| def import_snapshot(self, options: ImportOptions) -> ImportResult: | ||
| self._validate_snapshot_file(options.snapshot_file) | ||
| output_dir = self._resolve_output_dir(options.snapshot_file, options.output_dir) | ||
| db_path = self._backend.dump_to_db(options.snapshot_file, output_dir, options.device) | ||
|
|
||
| focus_state = None | ||
| if options.set_focus: | ||
| if self._focus_service is None: | ||
| self._focus_service = FocusService() | ||
| try: | ||
| focus_state = self._focus_service.set_project_focus( | ||
| db_path=db_path, | ||
| device_id=options.device, | ||
| ) | ||
| except (FocusFileInvalidError, DatabaseSchemaError) as exc: | ||
| # Focus write happens after the .db is on disk; if the new | ||
| # database fails validation (e.g. vendor produced a non-pt-snap | ||
| # schema), report it as an import execution failure rather than | ||
| # leaking the underlying domain error through the import API. | ||
| raise ImportExecutionError( | ||
| f"Imported database cannot be registered as focus: {exc}" | ||
| ) from exc | ||
|
|
||
| return ImportResult(db_path=db_path, device_id=options.device, focus_state=focus_state) | ||
|
|
||
| @staticmethod | ||
| def _validate_snapshot_file(path: Path) -> None: | ||
| if not path.exists(): | ||
| raise SnapshotFileInvalidError(f"Snapshot file does not exist: {path}") | ||
| if not path.is_file(): | ||
| raise SnapshotFileInvalidError(f"Snapshot path is not a file: {path}") | ||
| if path.suffix.lower() not in VALID_SUFFIXES: | ||
| raise SnapshotFileInvalidError( | ||
| f"Snapshot suffix must be .pkl or .pickle, got {path.suffix!r}" | ||
| ) | ||
|
Owner
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🟡 意见 4: 同一个 pickle 被读两遍:
对 11 MB fixture 没问题,但生产环境(GB 级 snapshot)会有不必要的内存峰值和延迟。vendor 的 建议: 移除 wrapper 的
Owner
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Addressed. The wrapper's |
||
|
|
||
| @staticmethod | ||
| def _resolve_output_dir(snapshot_file: Path, output_dir: Path | None) -> Path: | ||
| return output_dir.resolve() if output_dir is not None else snapshot_file.resolve().parent | ||
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🔴 Blocker 4:
[tool.setuptools.package-data]不包含 vendor 子树,需要验证 wheel 完整性pt_snap_cli = ["query/templates/*.yaml", "query/templates/*/*.yaml"]只显式列了 YAML 模板(本 hunk 之前的几行)。vendor 目录下的.py文件没被显式列出。虽然
[tool.setuptools.packages.find]设了include = ["pt_snap_cli*"],理论上会通过模块发现把 vendor 子包打进去 — 但这条路径没有任何 CI step 验证过。最危险的失败模式:
pip install -e .走 develop 模式,vendor 文件存在(本地 import 一切正常)pip install dist/*.whl装正式包,如果 setuptools.find 没把 vendor/... 收进去 →ImportToolMissingError在生产环境触发pytest走 develop 模式,这个问题永远抓不到强制要求: 在 CI 加 step:
pip install build python -m build --wheel pip install dist/*.whl --force-reinstall pt-snap import tests/fixtures/snapshots/snapshot_with_empty_cache.pkl --no-focusThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Addressed. The wheel smoke test is now wired into CI (only on Python 3.13 to keep the matrix cost down): it builds
dist/*.whl, force-reinstalls it, assertspt_snap_cli.vendor.memsnapdump.tools.adaptors.snapshot2db.run_dump_to_dbis importable, and runspt-snap importagainst the realsnapshot_with_empty_cache.pklfixture — verified locally to produce the same 11 MB.dbartifact. Any future packaging regression that drops the vendor closure will fail CI immediately instead of only surfacing in production.