Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 26 additions & 12 deletions docs/en/sharded-snapshotdb.md
Original file line number Diff line number Diff line change
Expand Up @@ -147,8 +147,9 @@ if a producer has discarded their identity; correct producer tracking is require
## Extensions, scopes and capabilities

Keep optional metadata, deduplicated stacks and ordered frames in isolated
`pt_snap_*` extension tables or sidecars, never append columns to positional base
tables. In particular, **do not attach an unprefixed native `callstack` table to
`pt_snap_*` extension tables or sidecars. Compatibility-v1 allows consumer columns
after the required base columns; their order, types and sole primary key must stay
unchanged. Native-v2 retains exact-column validation. In particular, **do not attach an unprefixed native `callstack` table to
inline v1**: the existing pt-snap layout detector correctly considers it a v1/v2
conflict. Native v2 remains a separate single-DB layout, not this compatibility
base. `pt_snap_metadata` and auxiliary reference tables are optional here.
Expand Down Expand Up @@ -357,11 +358,12 @@ sequentially.
Building, missing/inconsistent slices, unknown base versions and unsafe paths
are rejected before focus writes or query execution. Before **any** SQLite open,
both the bare P0 validator and resolver check **all** member/parent aliases and
live/dangling WAL, journal and SHM sidecars. Finalized, closed compatibility-v1
members use `mode=ro&immutable=1`, including cached query Context connections.
This accepts the fixed original producer's checkpointed, sidecar-free WAL headers
without creating sidecars or changing bytes. The immutable flag assumes a closed
finalized artifact, **not a live-WAL shortcut** or concurrent-writer guarantee.
WAL, journal and SHM transport. Compatibility-v1 members use SQLite locking and
`mode=ro` without `immutable=1`, including cached query Context connections.
Committed WAL data is read transactionally; alias sidecars and rollback journals
are rejected. A read-only open may create empty WAL/SHM protocol files in a writable
directory, but does not modify base rows. Concurrent checkpoint/publication may
be rejected safely; wait for the writer to settle and retry.
No repair, checkpoint, journal-mode change, migration or deletion occurs.
Native datasets retain persistent-WAL rejection; standalone Context defaults stay
`mode=ro` without immutable. Cache identity includes transport mode and generation.
Expand Down Expand Up @@ -507,17 +509,29 @@ additional ID-validation memory, not snapshot loading, replay, or total import R
`cacheHash = SHA256(b"mem_snapshot_parser_v2" + raw_pickle_bytes)`: the salt is
**prepended**. Independent `ptSnap.identity` fields are `sourceSha256`,
`requestedDevice`, `eventsPerSlice`, `outputFormat=compatibility-v1`,
`manifestContractVersion=1`, `exportContractVersion=1`, `referenceContractVersion=1`,
`manifestContractVersion=1`, `exportContractVersion=2`, `referenceContractVersion=1`,
`metadataSchemaVersion=1`, `importFormatVersion=1`, and full `targetRevision`.
The `ptSnap` object also contains `metadata`, `omittedDevices`, and `members`
(canonical relative path → finalized SHA256). Whole validated ownership/identity
(canonical relative path → logical parser-data SHA256 with `memberHashKind=parser-v1`
for new exports). Whole validated ownership/identity
requires every member/hash and identical per-slice metadata; unknown/external
caches and unrelated files are not adopted. CLI version alone does not invalidate.
caches and unrelated files are not claimed as owned. Valid Insight caches can be
reused read-only without producer metadata. CLI version alone does not invalidate.

Logical seals cover parser-owned base rows and their dictionary entries, not
consumer tables, indexes or appended columns. Both import and dataset queries
check declared seals; unsealed legacy artifacts receive structural, range,
dictionary-reference and `quick_check` validation only. Without a historical seal,
arbitrary changed values or deleted block rows cannot be proven unchanged.
Digests are not signatures. `quick_check` detects physical damage but does not
prove index/table equivalence against deliberate `writable_schema` tampering.
Default warm resolver reuse still reads all member bytes to verify the generation;
cold admission additionally scans rows. Validation time grows with dataset size.

Compatible publication is **no-replace even with force**. Identical recognized
pt-snap caches reuse without loading pickle (force does not bypass this rule).
Changed source path/content, options/contracts, corrupted members or GUI-modified
DBs require a new `--output-dir`; no repair, overwrite or deletion of the old
Changed source path/content, options/contracts, corrupted parser data or legacy
byte-attested DBs changed by a consumer require a new `--output-dir`; no repair, overwrite or deletion of the old
artifact occurs. Exclusive publication and focus compensation use the existing
checked transaction; source SHA256 is rechecked immediately before publish/reuse.
Native force replacement and recovery semantics are untouched.
Expand Down
24 changes: 16 additions & 8 deletions docs/zh/sharded-snapshotdb.md
Original file line number Diff line number Diff line change
Expand Up @@ -119,7 +119,8 @@ dictionary 的十进制整数 key 以文本存储,使用真实带设备后缀
## 扩展、scope 与能力

可选 metadata、去重 stack、有序 frame 使用隔离的 `pt_snap_*` 扩展表或 sidecar,
不能向按位置读取的基础表追加列。尤其**不能给内联 v1 附加未命名隔离的原生
兼容 v1 允许在必需基础列末尾追加消费者列,但基础列的顺序、类型和唯一主键不得改变;
原生 v2 仍严格校验完整列集合。尤其**不能给内联 v1 附加未命名隔离的原生
`callstack` 表**:现有布局检测会正确报告 v1/v2 冲突。原生 v2 仍是独立单库布局,
不是本兼容基础格式。这里不要求 `pt_snap_metadata` 或辅助引用表。

Expand Down Expand Up @@ -286,10 +287,10 @@ Context LRU 默认最多四个;数据集内容 generation 变化在下次 look

building、缺片/矛盾片、未知基础版本、不安全路径在 focus 写入或查询前拒绝。
**任何** SQLite 打开之前,裸 P0 validator 与 resolver 均先检查**全部**成员/父目录别名及
存活/悬空 WAL、journal、SHM sidecar。最终化、已关闭的兼容 v1 成员使用
`mode=ro&immutable=1`,缓存查询的 Context 连接也一样;接受固定原始生产端已 checkpoint、
无 sidecar 的 WAL header,不创建 sidecar、不改变字节。immutable 只适用于已关闭最终化
产物,**不是读取 live WAL 的捷径**,不保证并发 writer 安全。不修复、不 checkpoint、
WAL、journal、SHM 传输。兼容 v1 成员使用 SQLite 锁和不带 `immutable=1` 的
`mode=ro`,缓存查询的 Context 连接也一样;在事务中读取已提交的 WAL 数据,拒绝别名
sidecar 和回滚日志。只读打开可能在可写目录中创建空 WAL/SHM 协议文件,但不修改基础行。
并发 checkpoint/发布可能被安全拒绝,应等待写入稳定后重试。不修复、不 checkpoint、
不改变 journal mode、不迁移、不删除。原生数据集仍拒绝持久 WAL,单库 Context 默认仍
为不带 immutable 的 `mode=ro`。Cache 身份包含传输模式与 generation。这是只读校验,
不是文件系统沙箱或生产端锁。
Expand Down Expand Up @@ -404,14 +405,21 @@ raw-frame 丢失与字符串 OOM action 强制转换属于已知局限,不应

`cacheHash = SHA256(b"mem_snapshot_parser_v2" + 原始_pickle_字节)`,salt **前置**。
独立 `ptSnap.identity` 字段为 `sourceSha256`、`requestedDevice`、`eventsPerSlice`、
`outputFormat=compatibility-v1`、`manifestContractVersion=1`、`exportContractVersion=1`、
`outputFormat=compatibility-v1`、`manifestContractVersion=1`、`exportContractVersion=2`、
`referenceContractVersion=1`、`metadataSchemaVersion=1`、`importFormatVersion=1`、完整
`targetRevision`。`ptSnap` 另含 `metadata`、`omittedDevices` 和 `members`(规范相对路径
→ 最终化 SHA256)。整体已校验所有权/身份要求每个成员/哈希及逐片 metadata 相同;
→ 基础解析数据逻辑 SHA256,`memberHashKind=parser-v1`)。整体已校验所有权/身份要求每个成员/哈希及逐片 metadata 相同;
不接管未知/外部缓存或无关文件。CLI 版本变化本身不使缓存失效。

逻辑摘要覆盖解析器基础行及对应字典项,不覆盖消费者表、索引和追加列。导入和数据集查询
均校验已声明的摘要;无摘要的旧产物仅执行结构、事件范围、字典引用及 `quick_check`
校验。缺少历史摘要时,不能追溯证明任意字段未改动或 block 行未被删除。摘要不是签名,
`quick_check` 可检测物理损坏,但不能证明刻意使用 `writable_schema` 篡改后的索引与表
仍然一致。默认 resolver 热缓存仍读取全部成员字节核对 generation,冷准入还会扫描行,
因此校验耗时随数据量增长。

兼容发布 **force 也 no-replace**。已识别、身份相同的 pt-snap 缓存不加载 pickle 即复用,
force 不绕过该规则。源路径/内容、选项/合同变化、成员损坏或 GUI 修改 DB 后须换新
force 不绕过该规则。源路径/内容、选项/合同变化、基础解析数据损坏,或旧物理摘要产物被调用方修改后须换新
`--output-dir`,不修复、不覆盖、不删除旧产物。排他发布与 focus 补偿使用已有受检事务,
发布/复用前再次核对源 SHA256;原生 force 替换与恢复语义不变。

Expand Down
2 changes: 1 addition & 1 deletion src/pt_snap_cli/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -148,7 +148,7 @@
device_id=result.device_id,
reused=result.reused,
cache_miss_reason=result.cache_miss_reason,
metadata=asdict(result.metadata),
metadata=asdict(result.metadata) if result.metadata is not None else None,
focus_state=focus_state,
focus_source=result.focus_state.source if result.focus_state is not None else None,
**(
Expand Down Expand Up @@ -246,7 +246,7 @@
] = None,
) -> None:
"""PyTorch Memory Snapshot Analysis Tool."""
_JSON_MODE.set(False)

Check warning on line 249 in src/pt_snap_cli/cli.py

View workflow job for this annotation

GitHub Actions / Lint & Format

Result of call expression is of type "Token[bool]" and is not used; assign to variable "_" if this is intentional (reportUnusedCallResult)


@app.command("focus")
Expand Down Expand Up @@ -681,7 +681,7 @@
typer.echo(f"Devices: {info.devices}")
typer.echo(f"Dataset Support: {json.dumps(info.dataset_support, sort_keys=True)}")
typer.echo(
"Semantics Version: "

Check warning on line 684 in src/pt_snap_cli/cli.py

View workflow job for this annotation

GitHub Actions / Lint & Format

Implicit string concatenation not allowed (reportImplicitStringConcatenation)
f"{info.semantics_version if info.semantics_version is not None else 'none'}"
)
typer.echo()
Expand Down Expand Up @@ -1043,7 +1043,7 @@
try:
# Typer 0.27's non-standalone _main catches Exit and returns the
# code. Discarding that value made every domain _error() exit 0.
rv = app(standalone_mode=False)

Check warning on line 1046 in src/pt_snap_cli/cli.py

View workflow job for this annotation

GitHub Actions / Lint & Format

Type of "rv" is Any (reportAny)
code = rv if isinstance(rv, int) else 0
except ClickException as exc:
if _json_requested():
Expand Down
28 changes: 22 additions & 6 deletions src/pt_snap_cli/core/dataset_contract.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
from pathlib import Path
from typing import Literal, cast

from .dataset_files import require_readonly_member
from .dataset_files import readonly_member_uri, require_readonly_member
from .errors import DatabaseSchemaError
from .validation_budget import (
ValidationBudget,
Expand Down Expand Up @@ -344,13 +344,19 @@ def _safe_file(root: Path, relative: str) -> Path:


def _schema(
conn: sqlite3.Connection, table: str, columns: tuple[tuple[str, str], ...], loc: str
conn: sqlite3.Connection,
table: str,
columns: tuple[tuple[str, str], ...],
loc: str,
*,
allow_extra_columns: bool = False,
) -> None:
kind = conn.execute("SELECT type FROM sqlite_master WHERE name=?", (table,)).fetchone()
_need(kind == ("table",), loc, "table", f"{table} must be an actual table, not a view")
rows = conn.execute(f'PRAGMA table_info("{table}")').fetchall()
_need(
tuple((row[1], row[2].upper()) for row in rows) == columns,
tuple((row[1], row[2].upper()) for row in rows[: len(columns)]) == columns
and (allow_extra_columns or len(rows) == len(columns)),
loc,
"columns",
f"{table} column order/types differ from base contract",
Expand Down Expand Up @@ -390,7 +396,7 @@ def _validate_slice(
(block, BLOCK_COLUMNS),
("dictionary", DICTIONARY_COLUMNS),
):
_schema(conn, table, columns, loc)
_schema(conn, table, columns, loc, allow_extra_columns=True)
for table, column, values in (
(trace, "action", tuple(enumerate(ACTION_NAMES))),
(block, "state", BLOCK_STATES),
Expand Down Expand Up @@ -428,7 +434,8 @@ def _validate_slice(
).fetchone()
_need(bad is None, loc, "event_values", "invalid scalar type or negative byte metric")
boundary_count = conn.execute(f'SELECT COUNT(*) FROM "{trace}" WHERE id<0').fetchone()[0]
for row in checked_rows(conn.execute(f'SELECT * FROM "{block}"'), budget):
selected = ",".join(f'"{name}"' for name, _ in BLOCK_COLUMNS)
for row in checked_rows(conn.execute(f'SELECT {selected} FROM "{block}"'), budget):
block_id, address, size, requested, state, alloc, free = row
for name, value in zip(
("id", "address", "size", "requestedSize", "state", "allocEventId", "freeEventId"),
Expand Down Expand Up @@ -507,19 +514,28 @@ def validate_dataset(
with (
closing(
sqlite3.connect(
db_path.as_uri() + "?mode=ro&immutable=1",
readonly_member_uri(db_path),
uri=True,
timeout=sqlite_timeout(budget),
)
) as conn,
validation_progress(conn, budget),
):
conn.execute("PRAGMA query_only=ON")
conn.execute("BEGIN")
_need(
conn.execute("PRAGMA quick_check").fetchall() == [("ok",)],
item.file,
"integrity",
"SQLite member failed integrity check",
)
boundaries += _validate_slice(conn, device, item, identities, budget)
except sqlite3.DatabaseError as exc:
raise DatasetContractError(item.file, "sqlite", str(exc)) from exc
return DatasetValidation(
manifest, sum(item.event_count for item in manifest.devices), boundaries
)
except RecursionError as exc:
raise DatasetContractError("manifest.json", "read", "manifest nesting is too deep") from exc
except (OSError, UnicodeError, json.JSONDecodeError) as exc:
raise DatasetContractError("manifest.json", "read", str(exc)) from exc
43 changes: 36 additions & 7 deletions src/pt_snap_cli/core/dataset_files.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
"""Shared pre-open dataset checks. No repair, migration, WAL changes or SQLite opens."""
"""Shared read-only transport checks; never checkpoint or repair source databases."""

from __future__ import annotations

Expand All @@ -16,18 +16,47 @@ def require_readonly_member(root: Path, member: Path, *, immutable: bool = False
raise DatabaseSchemaError(f"Dataset member must not use symlink paths: {member}")
if member.resolve() != member or not member.is_file():
raise DatabaseSchemaError(f"Dataset member is missing or not a canonical file: {member}")
for suffix in ("-wal", "-journal", "-shm"):
sidecar = Path(str(member) + suffix)
if sidecar.exists() or sidecar.is_symlink():
raise DatabaseSchemaError(f"Dataset member has a live SQLite sidecar: {member}")
with member.open("rb") as source:
header = source.read(20)
# Only finalized compatibility datasets use immutable transport. Native
# datasets retain their non-WAL contract; no writer repair/checkpoint occurs.
wal = Path(str(member) + "-wal")
for suffix in ("-wal", "-journal", "-shm"):
sidecar = Path(str(member) + suffix)
if sidecar.is_symlink():
raise DatabaseSchemaError(f"Dataset member has an aliased SQLite sidecar: {member}")
if sidecar.exists():
if suffix == "-journal":
raise DatabaseSchemaError(
f"Dataset member has a rollback journal (live SQLite sidecar); "
f"a writer or recovery may be active, retry later: {member}"
)
if not immutable or header[18:20] != b"\x02\x02":
raise DatabaseSchemaError(f"Dataset member has a live SQLite sidecar: {member}")
if not sidecar.is_file() or not wal.is_file():
raise DatabaseSchemaError(f"Invalid SQLite WAL transport: {member}")
with wal.open("rb") as source:
magic = source.read(4)
if magic not in (b"", b"\x37\x7f\x06\x82", b"\x37\x7f\x06\x83"):
raise DatabaseSchemaError(f"Invalid SQLite WAL header: {member}")
# The immutable flag identifies finalized compatibility members here. The
# connection helper always uses SQLite locking for compatibility members.
# Native datasets retain their non-WAL contract; no repair/checkpoint occurs.
if not immutable and header[:16] == b"SQLite format 3\x00" and 2 in header[18:20]:
raise DatabaseSchemaError(f"Dataset member uses persistent WAL mode: {member}")


def member_is_immutable(member: Path, compatibility: bool = True) -> bool:
"""Sidecar absence cannot prove immutability: a consumer may start writing."""
return False


def readonly_member_uri(member: Path, compatibility: bool = True) -> str:
return (
member.as_uri()
+ "?mode=ro"
+ ("&immutable=1" if member_is_immutable(member, compatibility) else "")
)


def hash_file(path: Path, *, budget: ValidationBudget | None = None) -> str:
check_budget(budget)
digest = hashlib.sha256()
Expand Down
4 changes: 2 additions & 2 deletions src/pt_snap_cli/core/dataset_lifecycle_connection.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
import time

from pt_snap_cli.core.dataset_contract import SliceRecord
from pt_snap_cli.core.dataset_files import require_readonly_member
from pt_snap_cli.core.dataset_files import readonly_member_uri, require_readonly_member
from pt_snap_cli.core.dataset_sources import DatasetSourceResolver
from pt_snap_cli.core.errors import QueryExecutionError

Expand Down Expand Up @@ -48,7 +48,7 @@ def __enter__(self):
path = self.sources.dataset.root / item.file
immutable = self.sources.dataset.callstack_layout == "v1"
require_readonly_member(self.sources.dataset.root, path, immutable=immutable)
uri = path.as_uri() + "?mode=ro" + ("&immutable=1" if immutable else "")
uri = readonly_member_uri(path, immutable)
self.connection.execute(f'ATTACH DATABASE ? AS "s{item.index}"', [uri])
self.connection.execute("PRAGMA query_only=ON")
self.sources.budget.remaining()
Expand Down
4 changes: 2 additions & 2 deletions src/pt_snap_cli/core/dataset_lifecycle_scratch.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
from contextlib import contextmanager
from pathlib import Path

from pt_snap_cli.core.dataset_files import require_readonly_member
from pt_snap_cli.core.dataset_files import readonly_member_uri, require_readonly_member
from pt_snap_cli.core.dataset_lifecycle_connection import _AttachedSources
from pt_snap_cli.core.errors import QueryExecutionError
from pt_snap_cli.core.query_limits import MAX_SCRATCH_DB_BYTES, MIN_SCRATCH_DB_BYTES
Expand Down Expand Up @@ -50,7 +50,7 @@ def source(self, item):
path = self.sources.dataset.root / item.file
immutable = self.sources.dataset.callstack_layout == "v1"
require_readonly_member(self.sources.dataset.root, path, immutable=immutable)
uri = path.as_uri() + "?mode=ro" + ("&immutable=1" if immutable else "")
uri = readonly_member_uri(path, immutable)
self.execute("ATTACH DATABASE ? AS src", [uri])
try:
# Bound each source's independent page cache too. URI mode=ro is the
Expand Down
4 changes: 3 additions & 1 deletion src/pt_snap_cli/core/dataset_lifecycle_sql.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@
from contextlib import closing
from typing import cast

from pt_snap_cli.core.dataset_contract import BLOCK_COLUMNS
from pt_snap_cli.core.dataset_lifecycle_connection import _AttachedSources
from pt_snap_cli.core.errors import QueryExecutionError

Expand Down Expand Up @@ -98,8 +99,9 @@ def _large_observation_set(sources, items):
def _lifecycle_cte(sources, items, proof_items):
allocated = _proof(sources, proof_items, "allocEventId", 4)
completed = _proof(sources, proof_items, "freeEventId", 6)
selected = ",".join(f'b."{name}"' for name, _ in BLOCK_COLUMNS)
observations = " UNION ALL ".join(
f"SELECT b.*, {item.index} AS observation_slice_index, "
f"SELECT {selected}, {item.index} AS observation_slice_index, "
f"{allocated} AS allocation_proved, "
f"CASE WHEN {completed} THEN b.freeEventId END AS completed_free "
f'FROM "s{item.index}"."block_{sources.device.device_id}" b'
Expand Down
Loading
Loading