Skip to content

Latest commit

 

History

History
491 lines (381 loc) · 20.5 KB

database.md

File metadata and controls

491 lines (381 loc) · 20.5 KB

SnapshotDB Schema

中文 | English

Overview

SnapshotDB is the SQLite database format for persisting PyTorch memory profiling data. It supports multi-device (multi-GPU) snapshot storage and querying. Notably, it does more than simply converting raw pickle snapshot data into SQLite — it performs a complete "replay" of the raw memory snapshot data and records additional information during the replay, such as:

  1. The total memory pool size and total allocated block size after any event;
  2. The complete lifecycle of all memory blocks during the collection period (which event allocated them, which event freed them).

Example database file: snapshot.pkl.db


Producing a snapshot DB

pt-snap analyzes standalone SQLite SnapshotDB files and explicitly selected complete validated native/compatible datasets. Closed compatibility-v1 uses immutable read-only transport; native and standalone policies remain distinct. Acceptance, metadata/frame limits and measurements separate schema, source/lifecycle differential and pending GUI evidence. If you start with a raw PyTorch memory snapshot pickle, import the snapshot with the built-in backend:

Security warning: Import trusted pickle files only. Pickle deserialization can execute arbitrary code. The loader rejects non-builtins global objects, but pt-snap import is not a sandbox.

pt-snap import snapshot.pkl
pt-snap import snapshot.pkl --json

The import command uses pt-snap-cli's first-party snapshot runtime to produce snapshot.pkl.db next to the input file by default, then updates project focus so subsequent commands can use it directly:

pt-snap query --list

You can also generate the same SnapshotDB format with another compatible producer, then point pt-snap focus at the resulting .db file.

Import cache and metadata

Generated databases contain a pt_snap_metadata table with the source SHA-256, exact device selection, import format version, producer version, and completion time. pt-snap import reuses an existing target only when the full source SHA-256, import format version, and device selection match. Package version changes alone do not invalidate the cache.

# Inspect devices, trace bounds, and import-metadata status
pt-snap overview snapshot.pkl.db
pt-snap overview snapshot.pkl.db --json

# Inspect provenance and compatibility metadata
pt-snap metadata snapshot.pkl.db
pt-snap metadata snapshot.pkl.db --json

# Bypass a matching cache
pt-snap import snapshot.pkl --force

import --json reports db_path, device_id, reused, cache_miss_reason, metadata, focus_state, and top-level focus_source. metadata --json keeps its existing field names (db_path, status, reason, metadata) for compatibility.

Legacy or externally generated compatible databases without this table remain queryable for templates that do not require a recognized callstack layout. pt-snap detects the callstack layout from table columns (and treats pt_snap_metadata.import_format_version as an auxiliary check, not the only source of truth). Callstack templates use the matching SQL automatically.

Opening, focusing, and querying a database never writes the .db file and never migrates it in place. Import cache invalidation still rebuilds a destination when the current importer's format version does not match; that is separate from read compatibility. pt-snap metadata reports missing metadata as unavailable, and a later import of the same pickle rebuilds the database once before reuse.

Query workflow

Import replays allocator events and normalizes them into per-device trace_entry_<device> and block_<device> tables. Focus chooses a database and optional device; query templates then resolve those device-specific table names:

pt-snap focus snapshot.pkl.db --device 0
pt-snap query --template-use memory_peak
pt-snap query --template-use block --params '{"min_size": 1048576}'

Use pt-snap capabilities --json for the full template catalog, or pt-snap query --list and pt-snap query --template-info <name> to inspect the supported query surface. event, callstack_analysis, and active_memory_callstack_at_event work on both the current deduplicated callstack schema and older inline-text databases; see Callstack layout compatibility. See Querying for the complete workflow.


Database Structure

Tables

Table Description Records (example)
dictionary Enum mapping dictionary -
callstack Deduplicated callstack text shared by all devices 4,369
trace_entry_0 Device 0 event trace table 8,094
block_0 Device 0 memory block table -
pt_snap_metadata First-party import provenance and cache metadata 1

Naming convention: For multi-device scenarios, table names are suffixed with the device ID, e.g., trace_entry_1 and block_1 for device 1. The callstack table carries no device suffix because callstacks are shared across devices.


Table Definitions

1. trace_entry_{device} — Event Trace Table

Records complete trace information for memory management events.

Schema

CREATE TABLE trace_entry_0 (
    `id` INTEGER PRIMARY KEY,
    `action` INTEGER,
    `address` INTEGER,
    `size` INTEGER,
    `stream` INTEGER,
    `allocated` INTEGER,
    `active` INTEGER,
    `reserved` INTEGER,
    `callstackId` INTEGER
);

Columns

Column Type Constraints Description Example
id INTEGER PRIMARY KEY Unique event ID; negative values indicate system-generated events 1, -100
action INTEGER — Action type code (see below) 4
address INTEGER — Memory address 20697535234048
size INTEGER — Allocation size in bytes 41943040
stream INTEGER — CUDA/CANN stream ID 1276474240
allocated INTEGER — Total allocated bytes 136426496
active INTEGER — Total active bytes 136426496
reserved INTEGER — Memory pool reserved bytes 155189248
callstackId INTEGER NULL Reference to callstack.id; join to read the text 12

Action Type Codes

Value Name Description
0 segment_map Memory segment mapped (expandable segment)
1 segment_unmap Memory segment unmapped
2 segment_alloc Memory segment allocated
3 segment_free Memory segment freed
4 alloc Memory allocation event
5 free_requested Free request
6 free_completed Free completed
7 workspace_snapshot NPU-specific workspace memory pool snapshot

ID Constraints

Condition Constraint
id >= 0 Event IDs strictly follow the chronological order after snapshot collection starts, incrementing and unique
id < 0 Synthetic events generated from raw pickle data to reconstruct Segments that existed at snapshot collection start; event type is always segment_map or segment_alloc

Data Example

id=1, action=4, address=20697535234048, size=41943040, stream=1276474240
allocated=136426496, active=136426496, reserved=155189248
callstackId=12 -> callstack.callstack:
  /home/liuyekang/dev/projects/test/memory_leaks_demo.py:60 <module>
  /home/liuyekang/dev/projects/test/memory_leaks_demo.py:34 main
  /home/liuyekang/dev/projects/test/memory_leaks_demo.py:24 train
  /home/liuyekang/dev/projects/test/memory_leaks_demo.py:12 train_one_step

Reading Callstack Text

Join callstack to resolve callstackId:

SELECT t.id, t.size, c.callstack
FROM trace_entry_0 t
LEFT JOIN callstack c ON c.id = t.callstackId
WHERE t.id = 1;

2. block_{device} — Memory Block Table

Records detailed information about memory blocks and their lifecycle state.

Schema

CREATE TABLE block_0 (
    `id` INTEGER PRIMARY KEY,
    `address` INTEGER,
    `size` INTEGER,
    `requestedSize` INTEGER,
    `state` INTEGER DEFAULT 99,
    `allocEventId` INTEGER,
    `freeEventId` INTEGER
);

Columns

Column Type SQL default Description
id INTEGER — Unique block ID; dynamic blocks use their allocation event ID, while preexisting blocks use a negative ID
address INTEGER — Memory address
size INTEGER — Actual allocated size (includes alignment overhead)
requestedSize INTEGER — User-requested allocation size
state INTEGER 99 State code (see below)
allocEventId INTEGER — Associated allocation event ID; the producer writes -1 when it was not captured
freeEventId INTEGER — Associated free-completion event ID; the producer writes -1 when it was not captured

State Codes

Value Name Description
-1 inactive Inactive (freed)
0 active_pending_free Active, pending free
1 active_allocated Active, allocation
99 unknown Unknown (default)

ID Constraints

Condition Constraint
id >= 0 block.id matches allocEventId, pointing to the same-ID allocation event in trace_entry
id < 0 The block was already allocated before snapshot collection began. Only its existence and initial state are known from pickle data; allocation time is unknown. The negative value itself carries no semantic meaning beyond uniqueness

allocEventId Constraints

Condition Constraint
allocEventId >= 0 block.id matches allocEventId, pointing to the same-ID allocation event in trace_entry
allocEventId == -1 The allocation event for this block was not captured during snapshot collection (the block was already allocated before collection started)

freeEventId Constraints

Condition Constraint
freeEventId >= 0 Points to the corresponding free-completion event in trace_entry; it is independent of block.id
freeEventId == -1 The free completion event for this block was not captured during snapshot collection (the block was not freed when collection ended)

State Usage

block.state is only meaningful when block.id is negative; otherwise it has no practical use.

Size Calculation

block.size is the actual allocated size, while block.requestedSize is the user-requested size. Both values come from the snapshot producer or replayed allocator state. Their relationship depends on allocator configuration and must not be inferred with one universal alignment formula.

Data Example

id=-320, address=20697531023360, size=4194816, requestedSize=4194304
state=1, allocEventId=-1, freeEventId=-1

3. dictionary — Mapping Dictionary

Stores enum value mappings used to decode fields like action and state.

Schema

CREATE TABLE dictionary (
    `table` TEXT,     -- Table name
    `column` TEXT,    -- Column name
    `key` TEXT,       -- Encoded integer value (as string)
    `value` TEXT      -- Original string value
);

Columns

Column Type Description
table TEXT Parent table name, e.g., trace_entry_0, block_0
column TEXT Column name, e.g., action, state
key TEXT Encoded integer value as string, e.g., 4, 1
value TEXT Original string value, e.g., alloc, active_allocated

Data Example

table=trace_entry_0, column=action, key=4, value=alloc
table=block_0, column=state, key=1, value=active_allocated

4. callstack — Deduplicated Callstack Text

Snapshots repeat a small number of callstacks across a very large number of events, so the text is stored once per distinct callstack and referenced by trace_entry_<device>.callstackId.

Schema

CREATE TABLE callstack (
    `id` INTEGER PRIMARY KEY,
    `callstack` TEXT
);

Columns

Column Type Description
id INTEGER Callstack ID referenced by trace_entry_<device>.callstackId
callstack TEXT Callstack text, outermost frame first, one file:line name per line

Properties

Property Guarantee
Uniqueness One row per distinct ordered structured frame array; display text alone is not a lossless identity
Scope Shared by every device in the database; the table has no device suffix
Coverage Every event produced by import resolves to a row; callstackId is NULL only in externally generated databases

Current standalone pt-snap import retains this v2 text layout and adds structured frame tables (import_format_version = 3). Format 2 databases have the same text layout but no original frame records. Databases produced before callstack deduplication store inline callstack TEXT on each trace_entry_<device> table and have no shared callstack table (import_format_version = 1 when metadata is present).

Structured frames and ordered arrays (format 3)

frame stores each distinct original (filename, line, name) tuple once. callstack_frame associates a stack with an ordered array of frame IDs:

CREATE TABLE frame (
    id INTEGER PRIMARY KEY,
    filename TEXT NOT NULL,
    line INTEGER NOT NULL,
    name TEXT NOT NULL
);
CREATE TABLE callstack_frame (
    callstackId INTEGER NOT NULL,
    position INTEGER NOT NULL,
    frameId INTEGER NOT NULL
);
CREATE UNIQUE INDEX idx_callstack_frame_position
    ON callstack_frame (callstackId, position);

These tables are shared across devices. An event refers to the array through trace_entry_<device>.callstackId; read it in position ASC order to reproduce the original snapshot array. Position 0 is the innermost frame. The frame tree reverses traversal to start with the outermost caller. Recursive occurrences retain distinct positions even when they share a frame ID. Empty arrays have no association rows and a manifest frameCount of zero. Structured fields preserve colons, whitespace, Unicode and embedded newlines without parsing display strings.

Use event_frames to read an event's ordered frames and active_memory_frame_tree_at_event to obtain parent/child occupancy. See Frame memory breakdown. Text-only databases remain readable by existing queries, but these new queries require re-importing the original snapshot. Import detects the changed format and rebuilds the cache; read-only analysis never migrates a database.

Standalone frame capability boundary

Format 3 applies only to standalone imports. Native sharded imports remain format 2 and compatible exports remain format 1. A separate version-2 deduplicated frame extension supports dataset trees; see querying. callstack_frame_manifest(callstackId INTEGER PRIMARY KEY, frameCount INTEGER NOT NULL) records the expected length of every captured stack, including zero. Frame fields must be text filename/name and an integer line. event_frames returns no rows for absent, known-empty or corrupt arrays, never a partial reconstruction. Trees keep known-empty captured stacks separate from missing/corrupt evidence. Dropped tail/all links, noninteger positions and invalid field types fail validation. Read-only queries never repair stored evidence.

Callstack layout compatibility

pt-snap reads both layouts without rewriting the file:

Layout How to recognize it Callstack text
v2 (current import) trace_entry_<device>.callstackId plus shared callstack Join callstack.id
v1 (legacy) trace_entry_<device>.callstack TEXT, no callstackId, no shared table Read the column directly

event, callstack_analysis, and active_memory_callstack_at_event keep one template name, parameter set, and output schema. The query engine selects v1 or v2 SQL from the detected layout. Structured-frame queries additionally require the format 3 tables; their frame evidence is not reconstructed from legacy text.

Detection is read-only: focus, query, reports, and the Python API all open the database with SQLite mode=ro. Layout is not stored in .pt-snap/focus.json. Conflicting columns, mixed devices, a damaged callstack table, or metadata that disagrees with the physical schema leave the layout unrecognized and fail only callstack templates, instead of guessing or blocking focus, metadata, and other queries. There is no implicit v1→v2 migration.

pt-snap focus legacy.db
# Callstack layout: v1 (inline text)
pt-snap query --template-use event
pt-snap query --template-use callstack_analysis

5. pt_snap_metadata — Import Metadata Table

First-party imports create one metadata row for provenance and cache compatibility checks:

CREATE TABLE pt_snap_metadata (
    id INTEGER PRIMARY KEY CHECK (id = 1),
    metadata_schema_version INTEGER NOT NULL,
    import_format_version INTEGER NOT NULL,
    source_sha256 TEXT NOT NULL,
    source_size INTEGER NOT NULL,
    source_name TEXT NOT NULL,
    requested_device INTEGER,
    importer_name TEXT NOT NULL,
    importer_version TEXT NOT NULL,
    completed_at TEXT NOT NULL
);

Legacy and externally produced compatible databases may omit this table. They remain queryable, but pt-snap metadata reports their metadata as unavailable.

Published native datasets

import --events-per-slice N publishes a distinct pt-snap-native-v2 manifest, not a compatibility-v1 manifest or another standalone DB layout. Each finalized shard uses the existing v2 tables plus pt_snap_block_reference (version 1), with blockId INTEGER PRIMARY KEY, stream INTEGER NOT NULL, allocCallstack TEXT, freeCallstack TEXT. These are text lifecycle sources, not structured frames or an automatic cross-shard JOIN. pt_snap_metadata is required and identical in all members; native manifest identity additionally versions source/device/capacity, format/manifest/import/extension contracts, independently of CLI release version.

Real counts use positions per device, preserving original sparse IDs and native OOM/workspace events. Negative boundaries do not consume capacity. Reader validation includes full-member hashes and finalized identity/reference consistency before bounded event addressing; it never repairs or migrates files. Aggregate and lifecycle dataset queries remain unsupported. See the separate native manifest protocol and transaction/recovery guide. P0 compatibility-v1 table order, inline layout and manifest semantics are unchanged.

Explicit compatible export

import --format msinsight (alias compatibility-v1) publishes physical v1 trace_entry_<device> tables with the ninth column callstack TEXT, not views, with the exact positional block/dictionary schema. It removes staging's unprefixed callstack table and writes pt_snap_metadata.import_format_version=1, retaining isolated pt_snap_block_reference text lifecycle sources. Native datasets keep their text interning and v2 metadata, plus the separate version-2 deduplicated frame extension; standalone imports use format 3. Frames are never inferred from text.

Inline stack bytes scale with the sum of stack text over every event, not unique stacks; conversion temporarily holds native and inline tables and may leave SQLite free pages. Capacity does not bound blocks, loading RSS or total disk. Compatible exports additionally carry ptSnap.identity, metadata, omitted devices and a structural member inventory with empty digest values. Historical digest fields are accepted as inventories; compatible import reuse compares member names, not digest values, alongside source identity and required read-only structure. It does not scan arbitrary row contents, so structurally valid content edits can be reused. Recognized metadata does not attest database content integrity. Queries and explicit validators perform independent admission checks; external v1 datasets still need no metadata. Native dataset content verification is unchanged.

For finalized compatibility datasets ONLY, bare validation and cached analysis use mode=ro&immutable=1 after rejecting all live/dangling sidecars and aliases before the first SQLite open. Closed/checkpointed original WAL bytes/inventory remain unchanged. This assumes no active writer; native dataset non-WAL policy and standalone mode=ro defaults are unchanged. See compatible protocol and no-replace rules.