Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 18 additions & 2 deletions docs/en/database.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,9 @@

## Overview

SnapshotDB is the SQLite database format for persisting PyTorch memory profiling data. It supports multi-device (multi-GPU) snapshot storage and querying.
SnapshotDB is the SQLite database format for persisting PyTorch memory profiling data. It supports multi-device (multi-GPU) snapshot storage and querying. Notably, it does more than simply converting raw pickle snapshot data into SQLite — it performs a complete "replay" of the raw memory snapshot data and records additional information during the replay, such as:
1. The total memory pool size and total allocated block size after any event;
2. The complete lifecycle of all memory blocks during the collection period (which event allocated them, which event freed them).

**Example database file**: `snapshot_expandable.pkl.db`

Expand Down Expand Up @@ -71,7 +73,7 @@ CREATE TABLE trace_entry_0 (
| 4 | alloc | Memory allocation event |
| 5 | free_requested | Free request |
| 6 | free_completed | Free completed |
| 7 | workspace_snapshot | Workspace snapshot |
| 7 | workspace_snapshot | NPU-specific workspace memory pool snapshot |

#### ID Constraints

Expand Down Expand Up @@ -140,6 +142,20 @@ CREATE TABLE block_0 (
| `id >= 0` | `block.id` matches `allocEventId`, pointing to the same-ID allocation event in `trace_entry` |
| `id < 0` | The block was already allocated before snapshot collection began. Only its existence and initial state are known from pickle data; allocation time is unknown. The negative value itself carries no semantic meaning beyond uniqueness |

#### allocEventId Constraints

| Condition | Constraint |
|-----------|------------|
| `allocEventId >= 0` | `block.id` matches `allocEventId`, pointing to the same-ID allocation event in `trace_entry` |
| `allocEventId == -1` | The allocation event for this block was not captured during snapshot collection (the block was already allocated before collection started) |

#### freeEventId Constraints

| Condition | Constraint |
|-----------|------------|
| `freeEventId >= 0` | `block.id` matches `freeEventId`, pointing to the same-ID free event in `trace_entry` |
| `freeEventId == -1` | The free completion event for this block was not captured during snapshot collection (the block was not freed when collection ended) |

#### State Usage

`block.state` is only meaningful when `block.id` is negative; otherwise it has no practical use.
Expand Down
19 changes: 18 additions & 1 deletion docs/zh/database.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,9 @@
## 概述

SnapshotDB 是内存快照数据的 SQLite 数据库存储格式,用于持久化内存分配追踪信息。支持多设备(多 GPU)快照数据的存储和查询。
需要说明的是,它不仅仅是将内存快照原始pickle数据转成了SQLite数据库,而是针对原始内存快照数据进行了一次完整的“回放”,并记录了回放过程中的额外信息,如
1. 任一事件发生后的内存池总大小、总分配内存块大小;
2. 采集周期中,所有的内存块的完整生命周期(在哪个事件申请、哪个事件释放)

**数据库示例文件**: `snapshot_expandable.pkl.db`

Expand Down Expand Up @@ -71,7 +74,7 @@ CREATE TABLE trace_entry_0 (
| 4 | alloc | 内存分配事件 |
| 5 | free_requested | 释放请求 |
| 6 | free_completed | 释放完成 |
| 7 | workspace_snapshot | 工作区快照 |
| 7 | workspace_snapshot | NPU特有的workspace内存池快照 |

#### id 字段约束

Expand Down Expand Up @@ -140,6 +143,20 @@ CREATE TABLE block_0 (
| `id >= 0` | `block.id` 与 `allocEventId` 一致,共同指向 `trace_entry` 中相同 ID 的分配事件 |
| `id < 0` | 仅代表通过原始 pickle 数据中的 Segment 信息得知该内存块在采集开始时已分配,无从得知分配时间。负数值本身无实际含义,仅用于唯一标识 |

#### allocEventId约束

| 条件 | 约束 |
|------|------|
| `allocEventId >= 0` | `block.id` 与 `allocEventId` 一致,共同指向 `trace_entry` 中相同 ID 的分配事件 |
| `allocEventId == -1` | 代表在内存快照采集期间,未采集到该内存块的申请事件(在开始采集之前该内存块就已经申请分配完成) |

#### freeEventId约束

| 条件 | 约束 |
|------|------|
| `freeEventId >= 0` | `block.id` 与 `allocEventId` 一致,共同指向 `trace_entry` 中相同 ID 的分配事件 |
| `freeEventId == -1` | 代表在内存快照采集期间,未采集到该内存块的释放完成事件(在结束采集时该内存块并未释放) |

#### state 约束

`block.state` 仅在 `block.id` 为负数时才有实际意义,否则无实际用途。
Expand Down
Loading