Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -67,3 +67,25 @@ jobs:
name: coverage-report
path: coverage.xml
retention-days: 30

- name: Build wheel
if: matrix.python-version == '3.13'
run: |
python -m pip install --upgrade build
python -m build --wheel

- name: Wheel smoke test (verify vendored backend is packaged)
if: matrix.python-version == '3.13'
run: |
set -euo pipefail
python -m pip install --force-reinstall dist/*.whl
# Confirm the vendored closure was actually packaged into the wheel.
python -c "import pt_snap_cli.vendor.memsnapdump.tools.adaptors.snapshot2db as m; assert hasattr(m, 'run_dump_to_db'), 'vendored run_dump_to_db missing in installed wheel'"
# End-to-end import against a real fixture — catches any packaging
# regression where setuptools drops src/pt_snap_cli/vendor/.
FIXTURE=tests/fixtures/snapshots/snapshot_with_empty_cache.pkl
rm -rf /tmp/pt-snap-wheel-smoke
mkdir -p /tmp/pt-snap-wheel-smoke
pt-snap import "$FIXTURE" --no-focus --output-dir /tmp/pt-snap-wheel-smoke
test -f /tmp/pt-snap-wheel-smoke/snapshot_with_empty_cache.pkl.db
rm -rf /tmp/pt-snap-wheel-smoke
8 changes: 8 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,13 @@ pip install -e .

## Quick Start

To start from a raw PyTorch memory snapshot, import the pickle into a SnapshotDB with the built-in backend, then list the available query templates:

```bash
pt-snap import snapshot.pkl
pt-snap query --list
```

```bash
# Set the snapshot database and device
pt-snap focus examples/snapshot_expandable.pkl.db --device 0
Expand All @@ -33,6 +40,7 @@ See the [full quick start guide](docs/en/quickstart.md) for a walkthrough.
| Command | Description |
|---------|-------------|
| `pt-snap focus` | Set and manage analysis focus (database + device) |
| `pt-snap import <snapshot.pkl>` | Import a PyTorch memory snapshot pickle into a SnapshotDB |
| `pt-snap query` | Run memory analysis queries |
| `pt-snap config` | Manage global configuration |
| `pt-snap-mcp` | Start the MCP server for agent integration |
Expand Down
18 changes: 18 additions & 0 deletions docs/en/database.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,24 @@ SnapshotDB is the SQLite database format for persisting PyTorch memory profiling

---

## Producing a snapshot DB

`pt-snap` analyzes SQLite SnapshotDB files. If you start with a raw PyTorch memory snapshot pickle, import the snapshot with the built-in backend:

```bash
pt-snap import snapshot.pkl
```

The import command uses a vendored MemSnapDump backend to produce `snapshot.pkl.db` next to the input file by default, then updates project focus so subsequent commands can use it directly:

```bash
pt-snap query --list
```

You can also generate the same SnapshotDB format with another compatible producer, then point `pt-snap focus` at the resulting `.db` file.

---

## Database Structure

### Tables
Expand Down
9 changes: 9 additions & 0 deletions docs/en/quickstart.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,15 @@ pip install -e .

## Your First Analysis

### Optional: Import a PyTorch Snapshot

If you have a raw `.pkl` memory snapshot, import it first with the built-in backend:

```bash
pt-snap import snapshot.pkl
pt-snap query --list
```

### Step 1: Set the Snapshot Database and Device

Point `pt-snap` to your SQLite snapshot database file:
Expand Down
9 changes: 9 additions & 0 deletions docs/zh/quickstart.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,15 @@ pip install -e .

## 第一次分析

### 可选:导入 PyTorch 快照

如果你手上是原始 `.pkl` 内存快照,请先使用内建后端导入:

```bash
pt-snap import snapshot.pkl
pt-snap query --list
```

### 第一步:设置快照数据库和设备

将 `pt-snap` 指向你的 SQLite 快照数据库文件:
Expand Down
6 changes: 6 additions & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -67,10 +67,16 @@ pt_snap_cli = [
[tool.black]
line-length = 100
target-version = ["py310"]
extend-exclude = '''

@codeasier codeasier Jun 21, 2026 •

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Blocker 4: [tool.setuptools.package-data] 不包含 vendor 子树,需要验证 wheel 完整性

pt_snap_cli = ["query/templates/*.yaml", "query/templates/*/*.yaml"] 只显式列了 YAML 模板(本 hunk 之前的几行)。vendor 目录下的 .py 文件没被显式列出。

虽然 [tool.setuptools.packages.find] 设了 include = ["pt_snap_cli*"],理论上会通过模块发现把 vendor 子包打进去 — 但这条路径没有任何 CI step 验证过。

最危险的失败模式:

  • pip install -e . 走 develop 模式,vendor 文件存在(本地 import 一切正常)
  • pip install dist/*.whl 装正式包,如果 setuptools.find 没把 vendor/... 收进去 → ImportToolMissingError 在生产环境触发
  • 本地 CI pytest 走 develop 模式,这个问题永远抓不到

强制要求: 在 CI 加 step:

pip install build
python -m build --wheel
pip install dist/*.whl --force-reinstall
pt-snap import tests/fixtures/snapshots/snapshot_with_empty_cache.pkl --no-focus

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed. The wheel smoke test is now wired into CI (only on Python 3.13 to keep the matrix cost down): it builds dist/*.whl, force-reinstalls it, asserts pt_snap_cli.vendor.memsnapdump.tools.adaptors.snapshot2db.run_dump_to_db is importable, and runs pt-snap import against the real snapshot_with_empty_cache.pkl fixture — verified locally to produce the same 11 MB .db artifact. Any future packaging regression that drops the vendor closure will fail CI immediately instead of only surfacing in production.

(
^/src/pt_snap_cli/vendor/
)

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 black extend-exclude 配置正确: vendor 子树被排除在 black 重写之外,避免未来 vendor 同步时被反向风格化。这一条是 vendor 模式的关键防线。

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Acknowledged. extend-exclude is intentionally kept on both black and ruff so future re-syncs of the vendor closure do not get reformatted on every CI pass. ATTRIBUTION.md documents this contract.

'''

[tool.ruff]
line-length = 100
target-version = "py310"
extend-exclude = ["src/pt_snap_cli/vendor"]

[tool.ruff.lint]
select = ["E", "F", "W", "I", "N", "UP", "B", "C4"]

@codeasier codeasier Jun 21, 2026 •

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Blocker 3: vendor 子树缺少 LICENSE / NOTICE / 来源元数据

src/pt_snap_cli/vendor/memsnapdump/ 下 23 个 Python 文件原样复制自上游,但:

  • 没有 LICENSE-memsnapdump / NOTICE / ATTRIBUTION.md 标注来源
  • 没有上游 commit SHA / tag pin(本 PR 用什么版本?)
  • 没有上游 license 文本(Apache 2.0 / MIT?决定 pt-snap-cli 是否需要兼容声明)

强制要求: 在 src/pt_snap_cli/vendor/memsnapdump/ 下添加:

  • ATTRIBUTION.md: 写明上游仓库 URL + commit SHA + license + 同步策略
  • LICENSE: 上游 license 原文(项目 license 是 MIT,与上游协议需确认兼容)

CI 还可以加一个对账 step:

git log --oneline src/pt_snap_cli/vendor/memsnapdump/ | head -5
# 提醒:vendor 自上次同步后是否变化

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed. Added src/pt_snap_cli/vendor/memsnapdump/ATTRIBUTION.md (upstream URL, pinned SHA 87ea207372e0985790e1a28dab499dccf3c3b9a4, license status, sync strategy) and src/pt_snap_cli/vendor/memsnapdump/LICENSE (MIT, same as pt-snap-cli). The upstream MemSnapDump repo currently has no top-level LICENSE file — both projects share the same author and the closure is taken under the same MIT terms. When upstream publishes a formal LICENSE file, the SHA in ATTRIBUTION.md should be bumped and the upstream text should replace the local one.

Expand Down
34 changes: 34 additions & 0 deletions src/pt_snap_cli/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,12 +20,17 @@
FocusFileInvalidError,
FocusNotConfiguredError,
FocusService,
ImportExecutionError,
ImportOptions,
ImportService,
ImportToolMissingError,
InvalidCategoryError,
InvalidDeviceError,
PeakMemoryReport,
QueryExecutionError,
QueryService,
ReportService,
SnapshotFileInvalidError,
TemplateNotFoundError,
TemplateRenderError,
)
Expand Down Expand Up @@ -153,6 +158,35 @@ def focus_database(
_error(str(e))


@app.command("import")
def import_snapshot(
snapshot_file: Annotated[Path, typer.Argument(help="Path to .pkl snapshot")],
output_dir: Annotated[Path | None, typer.Option("--output-dir", "-o")] = None,
device: Annotated[int | None, typer.Option("--device", "-d")] = None,
no_focus: Annotated[bool, typer.Option("--no-focus", help="Skip focus update")] = False,
) -> None:
"""Import a PyTorch memory snapshot into a SQLite database."""
try:
result = ImportService().import_snapshot(
ImportOptions(
snapshot_file=snapshot_file,
output_dir=output_dir,
device=device,
set_focus=not no_focus,
)
)
except (
ImportToolMissingError,
ImportExecutionError,
SnapshotFileInvalidError,
) as e:
_error(str(e))

typer.echo(f"Imported: {result.db_path}")
if result.focus_state is not None and result.focus_state.focus_file is not None:
typer.echo(f"Focus: {result.focus_state.focus_file}")


@app.command("query")
def query_database(
db_path: Annotated[
Expand Down
8 changes: 8 additions & 0 deletions src/pt_snap_cli/context.py
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,14 @@ def device_ids(self) -> list[int]:
self._device_ids = self._discover_device_ids()
return self._device_ids

def discover_devices(self) -> list[int]:
"""Discover device IDs available in the database.

Method-style alias of the ``device_ids`` property, exposed for
callers (e.g. import service) that prefer an explicit method.
"""
return self.device_ids

def _discover_device_ids(self) -> list[int]:
"""Discover device IDs from database table names."""
device_ids = set()
Expand Down
12 changes: 12 additions & 0 deletions src/pt_snap_cli/core/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,16 +3,22 @@
DatabaseSchemaError,
FocusFileInvalidError,
FocusNotConfiguredError,
ImportExecutionError,
ImportToolMissingError,
InvalidCategoryError,
InvalidDeviceError,
PtSnapCoreError,
QueryExecutionError,
SnapshotFileInvalidError,
TemplateNotFoundError,
TemplateRenderError,
)
from pt_snap_cli.core.focus_service import FocusService
from pt_snap_cli.core.import_service import ImportService
from pt_snap_cli.core.models import (
FocusState,
ImportOptions,
ImportResult,
PeakMemoryReport,
QueryResult,
ResolvedFocus,
Expand All @@ -34,14 +40,20 @@
"TemplateNotFoundError",
"TemplateRenderError",
"QueryExecutionError",
"ImportToolMissingError",
"SnapshotFileInvalidError",
"ImportExecutionError",
"ResolvedFocus",
"FocusState",
"TemplateParameter",
"TemplateSummary",
"TemplateInfo",
"QueryResult",
"PeakMemoryReport",
"ImportOptions",
"ImportResult",
"FocusService",
"QueryService",
"ReportService",
"ImportService",
]
33 changes: 33 additions & 0 deletions src/pt_snap_cli/core/errors.py
Original file line number Diff line number Diff line change
Expand Up @@ -36,3 +36,36 @@ class TemplateRenderError(PtSnapCoreError):

class QueryExecutionError(PtSnapCoreError):
pass


class ImportToolMissingError(PtSnapCoreError):

@codeasier codeasier Jun 21, 2026 •

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 设计确认: ImportToolMissingError 的 docstring 把语义从"用户漏装 memsnapdump" 改为"vendor 打包异常",解释清楚。

但前面关于 _backend_available 死代码的意见会导致 ImportToolMissingError 在生产路径上实际上触发不到(除非 vendor import 失败 — 但那个分支在 snapshot_import_backend.py 的 try/except 里独立处理,不会到这里)。两个错误层抢同一类语义,实际只有一层会被触发。

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Acknowledged. The ImportToolMissingError semantics now match its docstring: it is only raised from snapshot_import_backend.py's ImportError branch (i.e. the vendored closure could not be imported — a packaging problem). With the dead _backend_available path removed, there is exactly one place that raises this error in production, not two competing ones.

"""Raised when the vendored snapshot import backend is unavailable.

In the vendored model, this no longer means "the user forgot to install
memsnapdump" — the dump2db backend is built in. It means the vendored
backend could not be imported or initialized, which is a packaging
or installation problem.
"""

pass


class SnapshotFileInvalidError(PtSnapCoreError):
"""Raised when a candidate snapshot pickle file cannot be used as input.

Triggers include: path does not exist, path is not a regular file,
suffix is not .pkl or .pickle, file cannot be unpickled, or the
unpickled top-level object is not a dict.
"""

pass


class ImportExecutionError(PtSnapCoreError):
"""Raised when the import backend itself fails to produce a database.

Covers upstream dump2db returning False, raising an exception,
or producing no .db artifact where one was expected.
"""

pass
70 changes: 70 additions & 0 deletions src/pt_snap_cli/core/import_service.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
from __future__ import annotations

import logging
from pathlib import Path

from pt_snap_cli.core.errors import (
DatabaseSchemaError,
FocusFileInvalidError,
ImportExecutionError,
SnapshotFileInvalidError,
)
from pt_snap_cli.core.focus_service import FocusService
from pt_snap_cli.core.models import ImportOptions, ImportResult
from pt_snap_cli.core.snapshot_import_backend import SnapshotImportBackend

logger = logging.getLogger(__name__)

VALID_SUFFIXES = {".pkl", ".pickle"}


class ImportService:
"""Import PyTorch snapshot pickle files through the vendored backend."""

def __init__(
self,
focus_service: FocusService | None = None,
backend: SnapshotImportBackend | None = None,
) -> None:
self._focus_service = focus_service
self._backend = backend or SnapshotImportBackend()

def import_snapshot(self, options: ImportOptions) -> ImportResult:
self._validate_snapshot_file(options.snapshot_file)
output_dir = self._resolve_output_dir(options.snapshot_file, options.output_dir)
db_path = self._backend.dump_to_db(options.snapshot_file, output_dir, options.device)

focus_state = None
if options.set_focus:
if self._focus_service is None:
self._focus_service = FocusService()
try:
focus_state = self._focus_service.set_project_focus(
db_path=db_path,
device_id=options.device,
)
except (FocusFileInvalidError, DatabaseSchemaError) as exc:
# Focus write happens after the .db is on disk; if the new
# database fails validation (e.g. vendor produced a non-pt-snap
# schema), report it as an import execution failure rather than
# leaking the underlying domain error through the import API.
raise ImportExecutionError(
f"Imported database cannot be registered as focus: {exc}"
) from exc

return ImportResult(db_path=db_path, device_id=options.device, focus_state=focus_state)

@staticmethod
def _validate_snapshot_file(path: Path) -> None:
if not path.exists():
raise SnapshotFileInvalidError(f"Snapshot file does not exist: {path}")
if not path.is_file():
raise SnapshotFileInvalidError(f"Snapshot path is not a file: {path}")
if path.suffix.lower() not in VALID_SUFFIXES:
raise SnapshotFileInvalidError(
f"Snapshot suffix must be .pkl or .pickle, got {path.suffix!r}"
)

@codeasier codeasier Jun 21, 2026 •

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 意见 4: pickle.load 在 backend 之前完整反序列化一次,backend 又会再 load 一次

同一个 pickle 被读两遍:

  • 第一次:pickle.load 完整反序列化(只为验证 isinstance(obj, dict))
  • 第二次:vendor dump() 内的 load_pickle_to_dict 又 pickle.load 一次

对 11 MB fixture 没问题,但生产环境(GB 级 snapshot)会有不必要的内存峰值和延迟。vendor 的 load_pickle_to_dict(file_util.py:62)已经做了 dict 类型校验,本层重复校验属于重复工作。

建议: 移除 wrapper 的 pickle.load,只校验存在 + 后缀 + 可读;或保留为快速 fail-fast,但把 is_dict 校验放到 backend 内部短路以避免重复反序列化。

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed. The wrapper's pickle.load is gone — _validate_snapshot_file now only checks existence, file-ness, and suffix. The vendored load_pickle_to_dict still raises a clear error on bad pickles, and that error propagates as ImportExecutionError from SnapshotImportBackend.dump_to_db. The duplicate full unpickle of multi-GB snapshots is avoided.


@staticmethod
def _resolve_output_dir(snapshot_file: Path, output_dir: Path | None) -> Path:
return output_dir.resolve() if output_dir is not None else snapshot_file.resolve().parent
36 changes: 36 additions & 0 deletions src/pt_snap_cli/core/models.py
Original file line number Diff line number Diff line change
Expand Up @@ -69,3 +69,39 @@ class PeakMemoryReport:
peak: dict[str, Any]
allocator_gap: dict[str, Any] | None
callstack_groups: list[dict[str, Any]]


@dataclass(frozen=True)
class ImportOptions:
"""Options for `pt-snap import`.

Attributes:
snapshot_file: Path to the input `.pkl` or `.pickle` snapshot.
output_dir: Directory where the generated `<name>.db` is written.
Defaults to the snapshot file's parent directory.
device: Optional device id to focus on. When None, all available
devices are imported.
set_focus: When True (default), also write project focus so subsequent
`pt-snap query` / `pt-snap report` commands reuse the new DB.
"""

snapshot_file: Path
output_dir: Path | None = None
device: int | None = None
set_focus: bool = True


@dataclass(frozen=True)
class ImportResult:
"""Result of `pt-snap import`.

Attributes:
db_path: Path to the generated SQLite database.
device_id: Device id used during import, or None when not specified.
focus_state: The FocusState written to project focus when
options.set_focus is True, otherwise None.
"""

db_path: Path
device_id: int | None
focus_state: FocusState | None
Loading
Loading