Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,10 @@ jobs:
FIXTURE=tests/fixtures/snapshots/snapshot_with_empty_cache.pkl
rm -rf /tmp/pt-snap-wheel-smoke
mkdir -p /tmp/pt-snap-wheel-smoke
DB=/tmp/pt-snap-wheel-smoke/snapshot_with_empty_cache.pkl.db
pt-snap import "$FIXTURE" --no-focus --output-dir /tmp/pt-snap-wheel-smoke
test -f /tmp/pt-snap-wheel-smoke/snapshot_with_empty_cache.pkl.db
test -f "$DB"
SECOND_IMPORT=$(pt-snap import "$FIXTURE" --no-focus --output-dir /tmp/pt-snap-wheel-smoke)
echo "$SECOND_IMPORT" | grep -q '^Reused:'
pt-snap metadata "$DB" --json | python -c "import json, sys; data=json.load(sys.stdin); assert data['status'] == 'available'"
rm -rf /tmp/pt-snap-wheel-smoke
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
**/superpowers/
10 changes: 7 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ To start from a raw PyTorch memory snapshot, import the pickle into a SnapshotDB

```bash
pt-snap import snapshot.pkl
pt-snap metadata snapshot.pkl.db
pt-snap query --list
```

Expand All @@ -41,7 +42,9 @@ See the [full quick start guide](docs/en/quickstart.md) for a walkthrough.
|---------|-------------|
| `pt-snap focus` | Set and manage analysis focus (database + device) |
| `pt-snap import <snapshot.pkl>` | Import a PyTorch memory snapshot pickle into a SnapshotDB |
| `pt-snap metadata [database.db]` | Inspect SnapshotDB import provenance and compatibility metadata |
| `pt-snap query` | Run memory analysis queries |
| `pt-snap report` | Generate higher-level memory analysis reports |
| `pt-snap config` | Manage global configuration |
| `pt-snap-mcp` | Start the MCP server for agent integration |

Expand All @@ -63,6 +66,7 @@ The server exposes the following tools:
| `list_templates` | List available query templates |
| `get_template_info` | Get template details and parameters |
| `execute_query` | Run a query template against the focused database |
| `get_database_metadata` | Inspect import metadata for the focused or specified database |

See the [MCP guide](docs/en/mcp.md) for setup and usage details.

Expand All @@ -79,11 +83,11 @@ See the [MCP guide](docs/en/mcp.md) for setup and usage details.

## Query Templates

6 built-in templates across 3 categories:
9 built-in templates across 3 categories:

- **Basic**: `block`, `event`, `allocation`
- **Statistical**: `callstack_analysis`, `memory_peak`
- **Business**: `leak_detection`
- **Statistical**: `active_blocks_at_event`, `allocator_gap`, `callstack_analysis`, `memory_peak`
- **Business**: `active_memory_callstack_at_event`, `leak_detection`

See [Querying](docs/en/querying.md) for details.

Expand Down
14 changes: 11 additions & 3 deletions README_zh.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,10 @@ pip install -e .
## 快速开始

```bash
# 从原始 pickle 导入;相同内容和配置会自动复用已有 DB
pt-snap import snapshot.pkl
pt-snap metadata snapshot.pkl.db

# 设置快照数据库和设备
pt-snap focus examples/snapshot_expandable.pkl.db --device 0

Expand All @@ -33,7 +37,10 @@ pt-snap query --template-use leak_detection --params '{"min_size": 1024}'
| 命令 | 说明 |
|------|------|
| `pt-snap focus` | 设置和管理分析焦点(数据库 + 设备) |
| `pt-snap import <snapshot.pkl>` | 将 PyTorch 原始内存快照导入 SnapshotDB |
| `pt-snap metadata [database.db]` | 查看 SnapshotDB 的导入来源与兼容性 metadata |
| `pt-snap query` | 运行内存分析查询 |
| `pt-snap report` | 生成高层内存分析报告 |
| `pt-snap config` | 管理全局配置 |
| `pt-snap-mcp` | 启动 MCP 服务器以支持 Agent 集成 |

Expand All @@ -55,6 +62,7 @@ pt-snap-mcp
| `list_templates` | 列出可用的查询模板 |
| `get_template_info` | 获取模板详情和参数 |
| `execute_query` | 对焦点数据库执行查询模板 |
| `get_database_metadata` | 查看焦点或指定数据库的导入 metadata |

详见 [MCP 指南](docs/zh/mcp.md)。

Expand All @@ -71,11 +79,11 @@ pt-snap-mcp

## 查询模板

6 个内置模板,3 个分类:
9 个内置模板,3 个分类:

- **Basic**: `block`, `event`, `allocation`
- **Statistical**: `callstack_analysis`, `memory_peak`
- **Business**: `leak_detection`
- **Statistical**: `active_blocks_at_event`, `allocator_gap`, `callstack_analysis`, `memory_peak`
- **Business**: `active_memory_callstack_at_event`, `leak_detection`

详见 [Querying](docs/zh/querying.md)。

Expand Down
2 changes: 2 additions & 0 deletions docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
| Quick Start | [en/quickstart.md](en/quickstart.md) |
| Context Management | [en/focus-management.md](en/focus-management.md) |
| Querying | [en/querying.md](en/querying.md) |
| MCP server | [en/mcp.md](en/mcp.md) |
| Database Schema | [en/database.md](en/database.md) |
| ResultMapper API | [en/result-mapper-api.md](en/result-mapper-api.md) |

Expand All @@ -17,5 +18,6 @@
| 快速开始 | [zh/quickstart.md](zh/quickstart.md) |
| Focus 管理 | [zh/focus-management.md](zh/focus-management.md) |
| 运行查询 | [zh/querying.md](zh/querying.md) |
| MCP 服务器 | [zh/mcp.md](zh/mcp.md) |
| 数据库格式 | [zh/database.md](zh/database.md) |
| ResultMapper API | [zh/result-mapper-api.md](zh/result-mapper-api.md) |
19 changes: 19 additions & 0 deletions docs/en/database.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,25 @@ pt-snap query --list

You can also generate the same SnapshotDB format with another compatible producer, then point `pt-snap focus` at the resulting `.db` file.

### Import cache and metadata

Generated databases contain a `pt_snap_metadata` table with the source SHA-256, exact device
selection, import format version, producer version, and completion time. `pt-snap import` reuses an
existing target only when the full source SHA-256, import format version, and device selection match.
Package version changes alone do not invalidate the cache.

```bash
# Inspect provenance and compatibility metadata
pt-snap metadata snapshot.pkl.db
pt-snap metadata snapshot.pkl.db --json

# Bypass a matching cache
pt-snap import snapshot.pkl --force
```

Legacy or externally generated compatible databases without this table remain queryable. Their
metadata status is reported as unavailable, and a later import rebuilds them once before reuse.

---

## Database Structure
Expand Down
6 changes: 6 additions & 0 deletions docs/en/mcp.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ This starts a FastMCP server that exposes tools for analyzing PyTorch memory sna
| `list_templates` | List available query templates, optionally filtered by category |
| `get_template_info` | Get detailed information about a template including parameters |
| `execute_query` | Execute a query template against the focused database |
| `get_database_metadata` | Inspect import provenance for the focused or specified database |

## Available Resources

Expand All @@ -48,6 +49,7 @@ This starts a FastMCP server that exposes tools for analyzing PyTorch memory sna
2. Call `list_templates` to discover available queries
3. Call `get_template_info` with a template name to see its parameters
4. Call `execute_query` with the template name and parameters
5. Call `get_database_metadata` when import provenance or cache compatibility is needed

## Example Usage

Expand All @@ -66,6 +68,10 @@ get_template_info("leak_detection")
# Run a query
execute_query("leak_detection", params={"min_size": 1024})
# Returns: {"total": 5, "returned": 5, "rows": [...]}

# Inspect database import metadata
get_database_metadata()
# Returns: {"status": "available", "metadata": {"source_sha256": "...", ...}}
```

## CLI Commands
Expand Down
9 changes: 9 additions & 0 deletions docs/en/quickstart.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,9 +18,18 @@ If you have a raw `.pkl` memory snapshot, import it first with the built-in back

```bash
pt-snap import snapshot.pkl
pt-snap metadata snapshot.pkl.db
pt-snap query --list
```

The import command stores a SHA-256 source fingerprint and import compatibility metadata inside the
generated database. Repeating the same import with the same device selection reuses the existing DB.
Use `--force` when you intentionally need to rebuild it:

```bash
pt-snap import snapshot.pkl --force
```

### Step 1: Set the Snapshot Database and Device

Point `pt-snap` to your SQLite snapshot database file:
Expand Down
21 changes: 21 additions & 0 deletions docs/zh/database.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,27 @@ SnapshotDB 是内存快照数据的 SQLite 数据库存储格式,用于持久

**数据库示例文件**: `snapshot_expandable.pkl.db`

## 生成数据库与导入缓存

如果输入是 PyTorch 原始 `.pkl` 内存快照,可以直接导入:

```bash
pt-snap import snapshot.pkl
```

新生成的数据库包含 `pt_snap_metadata` 表,记录原始文件 SHA-256、精确设备选择、导入格式
版本、生产者版本和完成时间。只有完整 SHA-256、导入格式版本和设备选择均匹配时,重复导入
才会复用已有 DB;仅升级 `pt-snap-cli` 包版本不会让缓存失效。

```bash
pt-snap metadata snapshot.pkl.db
pt-snap metadata snapshot.pkl.db --json
pt-snap import snapshot.pkl --force
```

旧版或外部生成且结构兼容的 DB 仍可正常查询。若没有 metadata,查询会返回 unavailable,
下一次执行对应 pickle 导入时会重建一次。

---

## 数据库结构
Expand Down
6 changes: 6 additions & 0 deletions docs/zh/mcp.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ pt-snap-mcp
| `list_templates` | 列出可用的查询模板,可按类别筛选 |
| `get_template_info` | 获取模板详情,包括参数信息 |
| `execute_query` | 对焦点数据库执行查询模板 |
| `get_database_metadata` | 查看焦点或指定数据库的导入来源 metadata |

## 可用资源

Expand All @@ -48,6 +49,7 @@ pt-snap-mcp
2. 调用 `list_templates` 发现可用查询
3. 调用 `get_template_info` 查看模板参数
4. 调用 `execute_query` 执行查询
5. 需要确认导入来源或缓存兼容性时调用 `get_database_metadata`

## 使用示例

Expand All @@ -66,6 +68,10 @@ get_template_info("leak_detection")
# 运行查询
execute_query("leak_detection", params={"min_size": 1024})
# 返回: {"total": 5, "returned": 5, "rows": [...]}

# 查看数据库导入 metadata
get_database_metadata()
# 返回: {"status": "available", "metadata": {"source_sha256": "...", ...}}
```

## CLI 命令
Expand Down
8 changes: 8 additions & 0 deletions docs/zh/quickstart.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,9 +18,17 @@ pip install -e .

```bash
pt-snap import snapshot.pkl
pt-snap metadata snapshot.pkl.db
pt-snap query --list
```

导入命令会在数据库内保存原始文件的 SHA-256 和导入兼容性 metadata。再次使用相同设备配置
导入相同内容时,会直接复用已有 DB。如需明确重建,可使用:

```bash
pt-snap import snapshot.pkl --force
```

### 第一步:设置快照数据库和设备

将 `pt-snap` 指向你的 SQLite 快照数据库文件:
Expand Down
17 changes: 17 additions & 0 deletions src/pt_snap_cli/api.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
DatabaseSchemaError,
FocusNotConfiguredError,
FocusService,
ImportMetadataService,
QueryService,
)

Expand Down Expand Up @@ -39,6 +40,7 @@ def __init__(
self._device_id = device_id
self._focus_service = FocusService(self._config)
self._query_service = QueryService(self._focus_service)
self._metadata_service = ImportMetadataService()

def get_focus(self) -> FocusState:
state = self._focus_service.get_focus(
Expand Down Expand Up @@ -121,3 +123,18 @@ def execute_query(
"device_id": result.device_id,
"rows": result.rows,
}

def get_database_metadata(self, db_path: str | None = None) -> dict[str, Any]:
resolved = self._focus_service.resolve_focus(
explicit_db_path=db_path if db_path is not None else self._db_path,
explicit_device_id=self._device_id,
)
if resolved.db_path is None:
raise RuntimeError("No database configured. Call set_focus() first.")
try:
inspection = self._metadata_service.inspect(resolved.db_path)
except DatabaseMissingError as exc:
raise FileNotFoundError(str(exc)) from exc
except DatabaseSchemaError as exc:
raise ValueError(str(exc)) from exc
return self._metadata_service.inspection_to_dict(inspection)
45 changes: 44 additions & 1 deletion src/pt_snap_cli/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@
FocusNotConfiguredError,
FocusService,
ImportExecutionError,
ImportMetadataService,
ImportOptions,
ImportService,
ImportToolMissingError,
Expand Down Expand Up @@ -164,6 +165,7 @@ def import_snapshot(
output_dir: Annotated[Path | None, typer.Option("--output-dir", "-o")] = None,
device: Annotated[int | None, typer.Option("--device", "-d")] = None,
no_focus: Annotated[bool, typer.Option("--no-focus", help="Skip focus update")] = False,
force: Annotated[bool, typer.Option("--force", help="Rebuild even when cache matches")] = False,
) -> None:
"""Import a PyTorch memory snapshot into a SQLite database."""
try:
Expand All @@ -173,6 +175,7 @@ def import_snapshot(
output_dir=output_dir,
device=device,
set_focus=not no_focus,
force=force,
)
)
except (
Expand All @@ -182,11 +185,51 @@ def import_snapshot(
) as e:
_error(str(e))

typer.echo(f"Imported: {result.db_path}")
action = "Reused" if result.reused else "Imported"
typer.echo(f"{action}: {result.db_path}")
if result.cache_miss_reason is not None:
typer.echo(f"Cache miss: {result.cache_miss_reason}")
if result.focus_state is not None and result.focus_state.focus_file is not None:
typer.echo(f"Focus: {result.focus_state.focus_file}")


@app.command("metadata")
def show_database_metadata(
db_path: Annotated[
Path | None, typer.Argument(help="Path to database file (optional if configured)")
] = None,
json_output: Annotated[bool, typer.Option("--json", help="Emit machine-readable JSON")] = False,
) -> None:
"""Show import metadata for a SnapshotDB."""
focus_service = _focus_service()
try:
resolved = focus_service.resolve_focus(explicit_db_path=db_path)
if resolved.db_path is None:
raise FocusNotConfiguredError("No database path specified and no database configured.")
service = ImportMetadataService()
inspection = service.inspect(resolved.db_path)
Comment thread
codeasier marked this conversation as resolved.
Resolved
except (
FocusFileInvalidError,
FocusNotConfiguredError,
DatabaseMissingError,
DatabaseSchemaError,
) as e:
_error(str(e))

if json_output:
typer.echo(json.dumps(service.inspection_to_dict(inspection), indent=2))
return

typer.echo(f"Database: {inspection.db_path}")
typer.echo(f"Status: {inspection.status}")
if inspection.reason is not None:
typer.echo(f"Reason: {inspection.reason}")
if inspection.metadata is not None:
typer.echo("Metadata:")
for key, value in cast(Mapping[str, object], asdict(inspection.metadata)).items():
typer.echo(f" {key}: {value}")


@app.command("query")
def query_database(
db_path: Annotated[
Expand Down
5 changes: 4 additions & 1 deletion src/pt_snap_cli/context.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
from collections.abc import Generator
from contextlib import contextmanager
from pathlib import Path
from urllib.parse import quote


class DatabaseNotFoundError(FileNotFoundError):
Expand Down Expand Up @@ -116,7 +117,9 @@ def connect(self) -> Generator[sqlite3.Connection, None, None]:
yield self._conn
return

uri = f"file:{self.db_path}?mode=ro"
# Keep the colon in Windows drive prefixes while encoding SQLite URI metacharacters.
encoded_path = quote(self.db_path.as_posix(), safe="/:")
Comment thread
codeasier marked this conversation as resolved.
Resolved
Comment thread
codeasier marked this conversation as resolved.
Resolved
uri = f"file:{encoded_path}?mode=ro"
self._conn = sqlite3.connect(uri, uri=True)
self._conn.row_factory = sqlite3.Row
try:
Expand Down
Loading
Loading