diff --git a/docs/en/sharded-snapshotdb.md b/docs/en/sharded-snapshotdb.md index 3dc0480..0bf64ec 100644 --- a/docs/en/sharded-snapshotdb.md +++ b/docs/en/sharded-snapshotdb.md @@ -147,8 +147,9 @@ if a producer has discarded their identity; correct producer tracking is require ## Extensions, scopes and capabilities Keep optional metadata, deduplicated stacks and ordered frames in isolated -`pt_snap_*` extension tables or sidecars, never append columns to positional base -tables. In particular, **do not attach an unprefixed native `callstack` table to +`pt_snap_*` extension tables or sidecars. Compatibility-v1 allows consumer columns +after the required base columns; their order, types and sole primary key must stay +unchanged. Native-v2 retains exact-column validation. In particular, **do not attach an unprefixed native `callstack` table to inline v1**: the existing pt-snap layout detector correctly considers it a v1/v2 conflict. Native v2 remains a separate single-DB layout, not this compatibility base. `pt_snap_metadata` and auxiliary reference tables are optional here. @@ -357,11 +358,12 @@ sequentially. Building, missing/inconsistent slices, unknown base versions and unsafe paths are rejected before focus writes or query execution. Before **any** SQLite open, both the bare P0 validator and resolver check **all** member/parent aliases and -live/dangling WAL, journal and SHM sidecars. Finalized, closed compatibility-v1 -members use `mode=ro&immutable=1`, including cached query Context connections. -This accepts the fixed original producer's checkpointed, sidecar-free WAL headers -without creating sidecars or changing bytes. The immutable flag assumes a closed -finalized artifact, **not a live-WAL shortcut** or concurrent-writer guarantee. +WAL, journal and SHM transport. Compatibility-v1 members use SQLite locking and +`mode=ro` without `immutable=1`, including cached query Context connections. +Committed WAL data is read transactionally; alias sidecars and rollback journals +are rejected. A read-only open may create empty WAL/SHM protocol files in a writable +directory, but does not modify base rows. Concurrent checkpoint/publication may +be rejected safely; wait for the writer to settle and retry. No repair, checkpoint, journal-mode change, migration or deletion occurs. Native datasets retain persistent-WAL rejection; standalone Context defaults stay `mode=ro` without immutable. Cache identity includes transport mode and generation. @@ -507,17 +509,29 @@ additional ID-validation memory, not snapshot loading, replay, or total import R `cacheHash = SHA256(b"mem_snapshot_parser_v2" + raw_pickle_bytes)`: the salt is **prepended**. Independent `ptSnap.identity` fields are `sourceSha256`, `requestedDevice`, `eventsPerSlice`, `outputFormat=compatibility-v1`, -`manifestContractVersion=1`, `exportContractVersion=1`, `referenceContractVersion=1`, +`manifestContractVersion=1`, `exportContractVersion=2`, `referenceContractVersion=1`, `metadataSchemaVersion=1`, `importFormatVersion=1`, and full `targetRevision`. The `ptSnap` object also contains `metadata`, `omittedDevices`, and `members` -(canonical relative path → finalized SHA256). Whole validated ownership/identity +(canonical relative path → logical parser-data SHA256 with `memberHashKind=parser-v1` +for new exports). Whole validated ownership/identity requires every member/hash and identical per-slice metadata; unknown/external -caches and unrelated files are not adopted. CLI version alone does not invalidate. +caches and unrelated files are not claimed as owned. Valid Insight caches can be +reused read-only without producer metadata. CLI version alone does not invalidate. + +Logical seals cover parser-owned base rows and their dictionary entries, not +consumer tables, indexes or appended columns. Both import and dataset queries +check declared seals; unsealed legacy artifacts receive structural, range, +dictionary-reference and `quick_check` validation only. Without a historical seal, +arbitrary changed values or deleted block rows cannot be proven unchanged. +Digests are not signatures. `quick_check` detects physical damage but does not +prove index/table equivalence against deliberate `writable_schema` tampering. +Default warm resolver reuse still reads all member bytes to verify the generation; +cold admission additionally scans rows. Validation time grows with dataset size. Compatible publication is **no-replace even with force**. Identical recognized pt-snap caches reuse without loading pickle (force does not bypass this rule). -Changed source path/content, options/contracts, corrupted members or GUI-modified -DBs require a new `--output-dir`; no repair, overwrite or deletion of the old +Changed source path/content, options/contracts, corrupted parser data or legacy +byte-attested DBs changed by a consumer require a new `--output-dir`; no repair, overwrite or deletion of the old artifact occurs. Exclusive publication and focus compensation use the existing checked transaction; source SHA256 is rechecked immediately before publish/reuse. Native force replacement and recovery semantics are untouched. diff --git a/docs/zh/sharded-snapshotdb.md b/docs/zh/sharded-snapshotdb.md index d608cc7..73ce759 100644 --- a/docs/zh/sharded-snapshotdb.md +++ b/docs/zh/sharded-snapshotdb.md @@ -119,7 +119,8 @@ dictionary 的十进制整数 key 以文本存储,使用真实带设备后缀 ## 扩展、scope 与能力 可选 metadata、去重 stack、有序 frame 使用隔离的 `pt_snap_*` 扩展表或 sidecar, -不能向按位置读取的基础表追加列。尤其**不能给内联 v1 附加未命名隔离的原生 +兼容 v1 允许在必需基础列末尾追加消费者列,但基础列的顺序、类型和唯一主键不得改变; +原生 v2 仍严格校验完整列集合。尤其**不能给内联 v1 附加未命名隔离的原生 `callstack` 表**:现有布局检测会正确报告 v1/v2 冲突。原生 v2 仍是独立单库布局, 不是本兼容基础格式。这里不要求 `pt_snap_metadata` 或辅助引用表。 @@ -286,10 +287,10 @@ Context LRU 默认最多四个;数据集内容 generation 变化在下次 look building、缺片/矛盾片、未知基础版本、不安全路径在 focus 写入或查询前拒绝。 **任何** SQLite 打开之前,裸 P0 validator 与 resolver 均先检查**全部**成员/父目录别名及 -存活/悬空 WAL、journal、SHM sidecar。最终化、已关闭的兼容 v1 成员使用 -`mode=ro&immutable=1`,缓存查询的 Context 连接也一样;接受固定原始生产端已 checkpoint、 -无 sidecar 的 WAL header,不创建 sidecar、不改变字节。immutable 只适用于已关闭最终化 -产物,**不是读取 live WAL 的捷径**,不保证并发 writer 安全。不修复、不 checkpoint、 +WAL、journal、SHM 传输。兼容 v1 成员使用 SQLite 锁和不带 `immutable=1` 的 +`mode=ro`,缓存查询的 Context 连接也一样;在事务中读取已提交的 WAL 数据,拒绝别名 +sidecar 和回滚日志。只读打开可能在可写目录中创建空 WAL/SHM 协议文件,但不修改基础行。 +并发 checkpoint/发布可能被安全拒绝,应等待写入稳定后重试。不修复、不 checkpoint、 不改变 journal mode、不迁移、不删除。原生数据集仍拒绝持久 WAL,单库 Context 默认仍 为不带 immutable 的 `mode=ro`。Cache 身份包含传输模式与 generation。这是只读校验, 不是文件系统沙箱或生产端锁。 @@ -404,14 +405,21 @@ raw-frame 丢失与字符串 OOM action 强制转换属于已知局限,不应 `cacheHash = SHA256(b"mem_snapshot_parser_v2" + 原始_pickle_字节)`,salt **前置**。 独立 `ptSnap.identity` 字段为 `sourceSha256`、`requestedDevice`、`eventsPerSlice`、 -`outputFormat=compatibility-v1`、`manifestContractVersion=1`、`exportContractVersion=1`、 +`outputFormat=compatibility-v1`、`manifestContractVersion=1`、`exportContractVersion=2`、 `referenceContractVersion=1`、`metadataSchemaVersion=1`、`importFormatVersion=1`、完整 `targetRevision`。`ptSnap` 另含 `metadata`、`omittedDevices` 和 `members`(规范相对路径 -→ 最终化 SHA256)。整体已校验所有权/身份要求每个成员/哈希及逐片 metadata 相同; +→ 基础解析数据逻辑 SHA256,`memberHashKind=parser-v1`)。整体已校验所有权/身份要求每个成员/哈希及逐片 metadata 相同; 不接管未知/外部缓存或无关文件。CLI 版本变化本身不使缓存失效。 +逻辑摘要覆盖解析器基础行及对应字典项,不覆盖消费者表、索引和追加列。导入和数据集查询 +均校验已声明的摘要;无摘要的旧产物仅执行结构、事件范围、字典引用及 `quick_check` +校验。缺少历史摘要时,不能追溯证明任意字段未改动或 block 行未被删除。摘要不是签名, +`quick_check` 可检测物理损坏,但不能证明刻意使用 `writable_schema` 篡改后的索引与表 +仍然一致。默认 resolver 热缓存仍读取全部成员字节核对 generation,冷准入还会扫描行, +因此校验耗时随数据量增长。 + 兼容发布 **force 也 no-replace**。已识别、身份相同的 pt-snap 缓存不加载 pickle 即复用, -force 不绕过该规则。源路径/内容、选项/合同变化、成员损坏或 GUI 修改 DB 后须换新 +force 不绕过该规则。源路径/内容、选项/合同变化、基础解析数据损坏,或旧物理摘要产物被调用方修改后须换新 `--output-dir`,不修复、不覆盖、不删除旧产物。排他发布与 focus 补偿使用已有受检事务, 发布/复用前再次核对源 SHA256;原生 force 替换与恢复语义不变。 diff --git a/src/pt_snap_cli/cli.py b/src/pt_snap_cli/cli.py index 43e5dd7..9222d02 100644 --- a/src/pt_snap_cli/cli.py +++ b/src/pt_snap_cli/cli.py @@ -148,7 +148,7 @@ def _import_json(result: ImportResult) -> dict[str, JsonValue]: device_id=result.device_id, reused=result.reused, cache_miss_reason=result.cache_miss_reason, - metadata=asdict(result.metadata), + metadata=asdict(result.metadata) if result.metadata is not None else None, focus_state=focus_state, focus_source=result.focus_state.source if result.focus_state is not None else None, **( diff --git a/src/pt_snap_cli/core/dataset_contract.py b/src/pt_snap_cli/core/dataset_contract.py index a8b03fd..ab8427d 100644 --- a/src/pt_snap_cli/core/dataset_contract.py +++ b/src/pt_snap_cli/core/dataset_contract.py @@ -16,7 +16,7 @@ from pathlib import Path from typing import Literal, cast -from .dataset_files import require_readonly_member +from .dataset_files import readonly_member_uri, require_readonly_member from .errors import DatabaseSchemaError from .validation_budget import ( ValidationBudget, @@ -344,13 +344,19 @@ def _safe_file(root: Path, relative: str) -> Path: def _schema( - conn: sqlite3.Connection, table: str, columns: tuple[tuple[str, str], ...], loc: str + conn: sqlite3.Connection, + table: str, + columns: tuple[tuple[str, str], ...], + loc: str, + *, + allow_extra_columns: bool = False, ) -> None: kind = conn.execute("SELECT type FROM sqlite_master WHERE name=?", (table,)).fetchone() _need(kind == ("table",), loc, "table", f"{table} must be an actual table, not a view") rows = conn.execute(f'PRAGMA table_info("{table}")').fetchall() _need( - tuple((row[1], row[2].upper()) for row in rows) == columns, + tuple((row[1], row[2].upper()) for row in rows[: len(columns)]) == columns + and (allow_extra_columns or len(rows) == len(columns)), loc, "columns", f"{table} column order/types differ from base contract", @@ -390,7 +396,7 @@ def _validate_slice( (block, BLOCK_COLUMNS), ("dictionary", DICTIONARY_COLUMNS), ): - _schema(conn, table, columns, loc) + _schema(conn, table, columns, loc, allow_extra_columns=True) for table, column, values in ( (trace, "action", tuple(enumerate(ACTION_NAMES))), (block, "state", BLOCK_STATES), @@ -428,7 +434,8 @@ def _validate_slice( ).fetchone() _need(bad is None, loc, "event_values", "invalid scalar type or negative byte metric") boundary_count = conn.execute(f'SELECT COUNT(*) FROM "{trace}" WHERE id<0').fetchone()[0] - for row in checked_rows(conn.execute(f'SELECT * FROM "{block}"'), budget): + selected = ",".join(f'"{name}"' for name, _ in BLOCK_COLUMNS) + for row in checked_rows(conn.execute(f'SELECT {selected} FROM "{block}"'), budget): block_id, address, size, requested, state, alloc, free = row for name, value in zip( ("id", "address", "size", "requestedSize", "state", "allocEventId", "freeEventId"), @@ -507,7 +514,7 @@ def validate_dataset( with ( closing( sqlite3.connect( - db_path.as_uri() + "?mode=ro&immutable=1", + readonly_member_uri(db_path), uri=True, timeout=sqlite_timeout(budget), ) @@ -515,11 +522,20 @@ def validate_dataset( validation_progress(conn, budget), ): conn.execute("PRAGMA query_only=ON") + conn.execute("BEGIN") + _need( + conn.execute("PRAGMA quick_check").fetchall() == [("ok",)], + item.file, + "integrity", + "SQLite member failed integrity check", + ) boundaries += _validate_slice(conn, device, item, identities, budget) except sqlite3.DatabaseError as exc: raise DatasetContractError(item.file, "sqlite", str(exc)) from exc return DatasetValidation( manifest, sum(item.event_count for item in manifest.devices), boundaries ) + except RecursionError as exc: + raise DatasetContractError("manifest.json", "read", "manifest nesting is too deep") from exc except (OSError, UnicodeError, json.JSONDecodeError) as exc: raise DatasetContractError("manifest.json", "read", str(exc)) from exc diff --git a/src/pt_snap_cli/core/dataset_files.py b/src/pt_snap_cli/core/dataset_files.py index 23f94b6..6fd59f8 100644 --- a/src/pt_snap_cli/core/dataset_files.py +++ b/src/pt_snap_cli/core/dataset_files.py @@ -1,4 +1,4 @@ -"""Shared pre-open dataset checks. No repair, migration, WAL changes or SQLite opens.""" +"""Shared read-only transport checks; never checkpoint or repair source databases.""" from __future__ import annotations @@ -16,18 +16,47 @@ def require_readonly_member(root: Path, member: Path, *, immutable: bool = False raise DatabaseSchemaError(f"Dataset member must not use symlink paths: {member}") if member.resolve() != member or not member.is_file(): raise DatabaseSchemaError(f"Dataset member is missing or not a canonical file: {member}") - for suffix in ("-wal", "-journal", "-shm"): - sidecar = Path(str(member) + suffix) - if sidecar.exists() or sidecar.is_symlink(): - raise DatabaseSchemaError(f"Dataset member has a live SQLite sidecar: {member}") with member.open("rb") as source: header = source.read(20) - # Only finalized compatibility datasets use immutable transport. Native - # datasets retain their non-WAL contract; no writer repair/checkpoint occurs. + wal = Path(str(member) + "-wal") + for suffix in ("-wal", "-journal", "-shm"): + sidecar = Path(str(member) + suffix) + if sidecar.is_symlink(): + raise DatabaseSchemaError(f"Dataset member has an aliased SQLite sidecar: {member}") + if sidecar.exists(): + if suffix == "-journal": + raise DatabaseSchemaError( + f"Dataset member has a rollback journal (live SQLite sidecar); " + f"a writer or recovery may be active, retry later: {member}" + ) + if not immutable or header[18:20] != b"\x02\x02": + raise DatabaseSchemaError(f"Dataset member has a live SQLite sidecar: {member}") + if not sidecar.is_file() or not wal.is_file(): + raise DatabaseSchemaError(f"Invalid SQLite WAL transport: {member}") + with wal.open("rb") as source: + magic = source.read(4) + if magic not in (b"", b"\x37\x7f\x06\x82", b"\x37\x7f\x06\x83"): + raise DatabaseSchemaError(f"Invalid SQLite WAL header: {member}") + # The immutable flag identifies finalized compatibility members here. The + # connection helper always uses SQLite locking for compatibility members. + # Native datasets retain their non-WAL contract; no repair/checkpoint occurs. if not immutable and header[:16] == b"SQLite format 3\x00" and 2 in header[18:20]: raise DatabaseSchemaError(f"Dataset member uses persistent WAL mode: {member}") +def member_is_immutable(member: Path, compatibility: bool = True) -> bool: + """Sidecar absence cannot prove immutability: a consumer may start writing.""" + return False + + +def readonly_member_uri(member: Path, compatibility: bool = True) -> str: + return ( + member.as_uri() + + "?mode=ro" + + ("&immutable=1" if member_is_immutable(member, compatibility) else "") + ) + + def hash_file(path: Path, *, budget: ValidationBudget | None = None) -> str: check_budget(budget) digest = hashlib.sha256() diff --git a/src/pt_snap_cli/core/dataset_lifecycle_connection.py b/src/pt_snap_cli/core/dataset_lifecycle_connection.py index 71beed0..2cf57b6 100644 --- a/src/pt_snap_cli/core/dataset_lifecycle_connection.py +++ b/src/pt_snap_cli/core/dataset_lifecycle_connection.py @@ -7,7 +7,7 @@ import time from pt_snap_cli.core.dataset_contract import SliceRecord -from pt_snap_cli.core.dataset_files import require_readonly_member +from pt_snap_cli.core.dataset_files import readonly_member_uri, require_readonly_member from pt_snap_cli.core.dataset_sources import DatasetSourceResolver from pt_snap_cli.core.errors import QueryExecutionError @@ -48,7 +48,7 @@ def __enter__(self): path = self.sources.dataset.root / item.file immutable = self.sources.dataset.callstack_layout == "v1" require_readonly_member(self.sources.dataset.root, path, immutable=immutable) - uri = path.as_uri() + "?mode=ro" + ("&immutable=1" if immutable else "") + uri = readonly_member_uri(path, immutable) self.connection.execute(f'ATTACH DATABASE ? AS "s{item.index}"', [uri]) self.connection.execute("PRAGMA query_only=ON") self.sources.budget.remaining() diff --git a/src/pt_snap_cli/core/dataset_lifecycle_scratch.py b/src/pt_snap_cli/core/dataset_lifecycle_scratch.py index 4a70fff..9f71521 100644 --- a/src/pt_snap_cli/core/dataset_lifecycle_scratch.py +++ b/src/pt_snap_cli/core/dataset_lifecycle_scratch.py @@ -14,7 +14,7 @@ from contextlib import contextmanager from pathlib import Path -from pt_snap_cli.core.dataset_files import require_readonly_member +from pt_snap_cli.core.dataset_files import readonly_member_uri, require_readonly_member from pt_snap_cli.core.dataset_lifecycle_connection import _AttachedSources from pt_snap_cli.core.errors import QueryExecutionError from pt_snap_cli.core.query_limits import MAX_SCRATCH_DB_BYTES, MIN_SCRATCH_DB_BYTES @@ -50,7 +50,7 @@ def source(self, item): path = self.sources.dataset.root / item.file immutable = self.sources.dataset.callstack_layout == "v1" require_readonly_member(self.sources.dataset.root, path, immutable=immutable) - uri = path.as_uri() + "?mode=ro" + ("&immutable=1" if immutable else "") + uri = readonly_member_uri(path, immutable) self.execute("ATTACH DATABASE ? AS src", [uri]) try: # Bound each source's independent page cache too. URI mode=ro is the diff --git a/src/pt_snap_cli/core/dataset_lifecycle_sql.py b/src/pt_snap_cli/core/dataset_lifecycle_sql.py index aa4e3d1..753d5dd 100644 --- a/src/pt_snap_cli/core/dataset_lifecycle_sql.py +++ b/src/pt_snap_cli/core/dataset_lifecycle_sql.py @@ -13,6 +13,7 @@ from contextlib import closing from typing import cast +from pt_snap_cli.core.dataset_contract import BLOCK_COLUMNS from pt_snap_cli.core.dataset_lifecycle_connection import _AttachedSources from pt_snap_cli.core.errors import QueryExecutionError @@ -98,8 +99,9 @@ def _large_observation_set(sources, items): def _lifecycle_cte(sources, items, proof_items): allocated = _proof(sources, proof_items, "allocEventId", 4) completed = _proof(sources, proof_items, "freeEventId", 6) + selected = ",".join(f'b."{name}"' for name, _ in BLOCK_COLUMNS) observations = " UNION ALL ".join( - f"SELECT b.*, {item.index} AS observation_slice_index, " + f"SELECT {selected}, {item.index} AS observation_slice_index, " f"{allocated} AS allocation_proved, " f"CASE WHEN {completed} THEN b.freeEventId END AS completed_free " f'FROM "s{item.index}"."block_{sources.device.device_id}" b' diff --git a/src/pt_snap_cli/core/dataset_resolver.py b/src/pt_snap_cli/core/dataset_resolver.py index 1f7ea7c..bf241a4 100644 --- a/src/pt_snap_cli/core/dataset_resolver.py +++ b/src/pt_snap_cli/core/dataset_resolver.py @@ -11,7 +11,9 @@ import hashlib import json import os +import sqlite3 from collections import OrderedDict +from contextlib import closing from dataclasses import dataclass from pathlib import Path from threading import RLock @@ -27,6 +29,7 @@ validate_dataset, ) from pt_snap_cli.core.dataset_files import hash_file as _hash_file +from pt_snap_cli.core.dataset_files import readonly_member_uri from pt_snap_cli.core.dataset_files import require_readonly_member as _require_readonly_member from pt_snap_cli.core.dataset_support import DATASET_SUPPORT from pt_snap_cli.core.errors import ( @@ -44,7 +47,14 @@ validate_native_dataset, validate_native_scope, ) -from pt_snap_cli.core.validation_budget import ValidationBudget, check_budget, checked_rows +from pt_snap_cli.core.snapshot_cache import validate_cache +from pt_snap_cli.core.validation_budget import ( + ValidationBudget, + check_budget, + checked_rows, + sqlite_timeout, + validation_progress, +) _FileStamp = tuple[int, int, int, int, int, int, int] _Generation = tuple[tuple[str, _FileStamp], ...] @@ -215,10 +225,38 @@ def _generation( root, member, immutable=not isinstance(manifest, NativeManifest) ) stamps.append((item.file, _stamp(member))) + wal = Path(str(member) + "-wal") + if wal.exists(): + stamps.append((item.file + "-wal", _stamp(wal))) check_budget(budget) return tuple(stamps) +def _prepare_compatibility_transport( + root: Path, manifest: DatasetManifest, budget: ValidationBudget | None +) -> None: + """Establish read-only WAL sidecars before capturing physical change tokens.""" + for device in checked_rows(manifest.devices, budget): + for item in checked_rows(device.slices, budget): + member = root / item.file + with member.open("rb") as source: + header = source.read(20) + if header[18:20] != b"\x02\x02" or Path(str(member) + "-wal").exists(): + continue + try: + with ( + closing( + sqlite3.connect( + readonly_member_uri(member), uri=True, timeout=sqlite_timeout(budget) + ) + ) as conn, + validation_progress(conn, budget), + ): + conn.execute("SELECT name FROM sqlite_master LIMIT 1").fetchone() + except sqlite3.DatabaseError as exc: + raise DatabaseSchemaError(f"Cannot open dataset member: {exc}") from exc + + def _cacheable(generation: _Generation) -> bool: # This is a caller contract, not an automatic inference from OS or filesystem # metadata. Windows ctime historically means creation time, so stay strong. @@ -245,6 +283,10 @@ def _content_fingerprint( for item in checked_rows(device.slices, budget): digest.update(item.file.encode("utf-8")) digest.update(_hash_file(root / item.file, budget=budget).encode("ascii")) + wal = root / (item.file + "-wal") + if wal.exists(): + digest.update(b"-wal") + digest.update(_hash_file(wal, budget=budget).encode("ascii")) return digest.hexdigest() @@ -325,6 +367,12 @@ def inspect( # Preflight every member before the first SQLite open. Capturing # both ends also refuses publication during validation or hashing. generation = _generation(root, planned, budget) + if not isinstance(planned, NativeManifest): + # Read-only SQLite may create WAL/SHM for a closed WAL-mode DB. + # Do this only after all paths pass preflight, then capture the + # actual transport. Subsequent content changes still fail below. + _prepare_compatibility_transport(root, planned, budget) + generation = _generation(root, planned, budget) if manifest_stamp != _stamp(manifest_path): raise DatabaseSchemaError("Dataset manifest changed during inspection; retry.") # Compatibility manifests contain no shard digests. Bracket row @@ -340,6 +388,33 @@ def inspect( if isinstance(planned, NativeManifest) else validate_dataset(root, budget=budget) ) + if ( + not isinstance(planned, NativeManifest) + and isinstance(raw, dict) + and ("parserCache" in raw or "ptSnap" in raw) + ): + try: + attestation = raw.get("parserCache", {}) + if not isinstance(attestation, dict): + raise ValueError("parserCache: expected object") + producer = raw.get("ptSnap", {}) + if not isinstance(producer, dict): + raise ValueError("ptSnap: expected object") + identity = producer.get("identity", {}) + if not isinstance(identity, dict): + raise ValueError("ptSnap.identity: expected object") + selected = attestation.get("requestedDevice") + if selected is None: + selected = identity.get("requestedDevice") + if selected is not None and (type(selected) is not int or selected < 0): + raise ValueError( + "parserCache.requestedDevice: expected nonnegative integer" + ) + validate_cache(root, raw["cacheHash"], selected, budget=budget) + except (ValueError, KeyError, TypeError, sqlite3.DatabaseError) as exc: + raise DatasetContractError( + "parserCache" if "parserCache" in raw else "ptSnap", "seal", str(exc) + ) from exc digest = hashlib.sha256() digest.update(before.encode("ascii")) for device in checked_rows(validation.manifest.devices, budget): @@ -353,6 +428,10 @@ def inspect( else _hash_file(root / item.file, budget=budget) ) digest.update(content_hash.encode("ascii")) + wal = root / (item.file + "-wal") + if wal.exists(): + digest.update(b"-wal") + digest.update(_hash_file(wal, budget=budget).encode("ascii")) fingerprint = digest.hexdigest() if compatibility_before is not None and compatibility_before != fingerprint: raise DatabaseSchemaError("Dataset content changed during inspection; retry.") @@ -383,10 +462,19 @@ def inspect( while len(_VALIDATED) > _CACHE_LIMIT: _ = _VALIDATED.popitem(last=False) return result - except (DatasetContractError, OSError, UnicodeError, json.JSONDecodeError) as exc: + except ( + DatasetContractError, + OSError, + UnicodeError, + json.JSONDecodeError, + ) as exc: with _CACHE_LOCK: _ = _VALIDATED.pop(cache_key, None) raise DatabaseSchemaError(f"Invalid dataset: {exc}") from exc + except RecursionError as exc: + with _CACHE_LOCK: + _ = _VALIDATED.pop(cache_key, None) + raise DatabaseSchemaError("Invalid dataset: manifest nesting is too deep") from exc except (DatabaseSchemaError, QueryTimeoutError): with _CACHE_LOCK: _ = _VALIDATED.pop(cache_key, None) diff --git a/src/pt_snap_cli/core/dataset_sources.py b/src/pt_snap_cli/core/dataset_sources.py index 68a4d0a..550dfb8 100644 --- a/src/pt_snap_cli/core/dataset_sources.py +++ b/src/pt_snap_cli/core/dataset_sources.py @@ -21,7 +21,7 @@ from pt_snap_cli.context import DatabaseNotFoundError, SchemaVersionError from pt_snap_cli.core.context_cache import ContextCache from pt_snap_cli.core.dataset_contract import SliceRecord, _unique_object -from pt_snap_cli.core.dataset_files import require_readonly_member +from pt_snap_cli.core.dataset_files import member_is_immutable, require_readonly_member from pt_snap_cli.core.dataset_resolver import ResolvedDataset from pt_snap_cli.core.errors import ( DatabaseSchemaError, @@ -224,7 +224,7 @@ def read( context = self.cache.get( path, generation=self.dataset.fingerprint, - immutable=self.dataset.callstack_layout == "v1", + immutable=member_is_immutable(path, self.dataset.callstack_layout == "v1"), ) self.query_count += 1 return QueryExecutor(context).execute( diff --git a/src/pt_snap_cli/core/focus_service.py b/src/pt_snap_cli/core/focus_service.py index 23e0b0b..f6fd29c 100644 --- a/src/pt_snap_cli/core/focus_service.py +++ b/src/pt_snap_cli/core/focus_service.py @@ -96,7 +96,11 @@ def _checkpoint_file(path: Path, content: bytes, mode: int) -> tuple[Path, tuple if written <= 0: raise OSError("Could not write focus recovery checkpoint") remaining = remaining[written:] - os.fchmod(fd, mode) + if hasattr(os, "fchmod"): + os.fchmod(fd, mode) + else: + FocusService._verify_checkpoint(backup, (identity.st_dev, identity.st_ino)) + os.chmod(backup, mode) os.fsync(fd) FocusService._verify_checkpoint(backup, (identity.st_dev, identity.st_ino)) except BaseException: diff --git a/src/pt_snap_cli/core/import_metadata.py b/src/pt_snap_cli/core/import_metadata.py index e713803..0d95f06 100644 --- a/src/pt_snap_cli/core/import_metadata.py +++ b/src/pt_snap_cli/core/import_metadata.py @@ -1,7 +1,6 @@ from __future__ import annotations import hashlib -import json import sqlite3 from contextlib import closing from dataclasses import asdict @@ -24,6 +23,7 @@ ) from pt_snap_cli.core.msinsight_export import inspect_owned_cache from pt_snap_cli.core.native_dataset_contract import NativeManifest +from pt_snap_cli.core.snapshot_cache import parse_cache_manifest METADATA_TABLE = "pt_snap_metadata" METADATA_SCHEMA_VERSION = 1 @@ -91,8 +91,10 @@ def inspect(self, db_path: Path | str) -> MetadataInspection: return MetadataInspection(path, "available", dataset.validation.manifest.metadata) # External P0 metadata remains optional. Only a recognized complete # pt-snap manifest attestation can provide dataset-wide metadata. - with (dataset.root / "manifest.json").open(encoding="utf-8") as source: - raw = json.load(source) + try: + raw = parse_cache_manifest((dataset.root / "manifest.json").read_bytes()) + except (OSError, ValueError): + return MetadataInspection(path, "invalid", reason="metadata_invalid") if "ptSnap" in raw: try: owned = inspect_owned_cache(dataset.root) diff --git a/src/pt_snap_cli/core/import_service.py b/src/pt_snap_cli/core/import_service.py index 0f5722d..8f5df42 100644 --- a/src/pt_snap_cli/core/import_service.py +++ b/src/pt_snap_cli/core/import_service.py @@ -7,6 +7,7 @@ from dataclasses import replace from pathlib import Path +from pt_snap_cli.core.dataset_contract import parse_manifest, validate_dataset from pt_snap_cli.core.dataset_import_backend import DatasetImportBackend from pt_snap_cli.core.dataset_resolver import DatasetResolver, _require_readonly_member from pt_snap_cli.core.errors import ( @@ -36,7 +37,6 @@ compatible_identity, convert_shards, inspect_owned_cache, - salted_hash, ) from pt_snap_cli.core.native_dataset_contract import ( NATIVE_FORMAT, @@ -44,6 +44,7 @@ cache_identity, ) from pt_snap_cli.core.sharded_replay_service import ShardedReplayResult +from pt_snap_cli.core.snapshot_cache import parse_cache_manifest, source_hashes, validate_cache from pt_snap_cli.core.snapshot_import_backend import SnapshotImportBackend logger = logging.getLogger(__name__) @@ -289,13 +290,42 @@ def _import_compatible_dataset(self, options: ImportOptions) -> ImportResult: target = self._msinsight_backend.target_path(options.snapshot_file, output) if target.resolve() != target or target.is_symlink(): raise ImportExecutionError("Compatible output must be canonical and non-symlink.") - source_hash = self._hash_source(options.snapshot_file) try: - cache_hash = salted_hash(options.snapshot_file) - except OSError as exc: + source_hash, cache_hash = source_hashes(options.snapshot_file) + except (OSError, ValueError) as exc: raise ImportExecutionError(f"Cannot calculate msinsight cacheHash: {exc}") from exc if target.exists(): try: + external_manifest = parse_cache_manifest((target / "manifest.json").read_bytes()) + if isinstance(external_manifest, dict) and "ptSnap" not in external_manifest: + # A compatible producer is not pt-snap-owned. Reuse read-only; + # never invent provenance metadata or replace it under force. + checked = validate_cache(target, cache_hash, options.device) + manifest = parse_manifest(checked) + if ( + options.events_per_slice is not None + and options.events_per_slice != manifest.events_per_slice + ): + raise ValueError( + "Explicit events_per_slice does not match the existing compatible " + "cache; choose a new output directory." + ) + # A self-consistent seal is not proof of semantic validity. + # Use shared P0 without repeating the resolver's seal scan. + validate_dataset(target) + self._recheck_source(options.snapshot_file, source_hash) + return ImportResult( + target, + options.device, + self._set_focus_if_requested(target, options), + True, + None, + None, + target, + tuple(sorted(d.device_id for d in manifest.devices)), + sum(len(d.slices) for d in manifest.devices), + COMPATIBILITY_FORMAT, + ) owned = inspect_owned_cache(target) if ( owned.identity != compatible_identity(source_hash, options.device, capacity) @@ -304,6 +334,11 @@ def _import_compatible_dataset(self, options: ImportOptions) -> ImportResult: or owned.metadata.source_size != options.snapshot_file.stat().st_size ): raise ValueError("Source/options/format identity changed") + except FileNotFoundError as exc: + raise ImportExecutionError( + f"Existing compatible output is not reusable; preserved even under --force. " + f"Required artifact file is missing: {exc.filename}. Choose a new output directory." + ) from exc except ( ValueError, KeyError, diff --git a/src/pt_snap_cli/core/models.py b/src/pt_snap_cli/core/models.py index e994284..d7c7da5 100644 --- a/src/pt_snap_cli/core/models.py +++ b/src/pt_snap_cli/core/models.py @@ -207,7 +207,7 @@ class ImportResult: device_id: int | None focus_state: FocusState | None reused: bool - metadata: ImportMetadata + metadata: ImportMetadata | None cache_miss_reason: CacheMissReason | None dataset_path: Path | None = None devices: tuple[int, ...] = () diff --git a/src/pt_snap_cli/core/msinsight_export.py b/src/pt_snap_cli/core/msinsight_export.py index e1764af..6ff24e2 100644 --- a/src/pt_snap_cli/core/msinsight_export.py +++ b/src/pt_snap_cli/core/msinsight_export.py @@ -10,12 +10,11 @@ from __future__ import annotations -import hashlib -import json import sqlite3 from contextlib import closing from dataclasses import asdict, dataclass from pathlib import Path +from typing import Any from .dataset_contract import ( ACTION_NAMES, @@ -24,30 +23,25 @@ TRACE_COLUMNS, DatasetContractError, DatasetManifest, - _unique_object, validate_dataset, ) -from .dataset_files import hash_file, require_readonly_member +from .dataset_files import hash_file, readonly_member_uri, require_readonly_member from .dataset_import_backend import DatasetImportBackend from .errors import ImportExecutionError from .import_metadata_contract import metadata_from_mapping from .models import ImportMetadata from .sharded_replay_service import ShardedReplayResult +from .snapshot_cache import parse_cache_manifest, seal_manifest, source_hashes, validate_cache COMPATIBILITY_FORMAT = "compatibility-v1" DEFAULT_CAPACITY = 500000 -SALT = b"mem_snapshot_parser_v2" # Independent semantic versions, not the installed CLI version or GUI cacheHash. -EXPORT_VERSION = 1 +EXPORT_VERSION = 2 REFERENCE_VERSION = 1 def salted_hash(source: Path) -> str: - digest = hashlib.sha256(SALT) - with source.open("rb") as file: - for chunk in iter(lambda: file.read(1024 * 1024), b""): - digest.update(chunk) - return digest.hexdigest() + return source_hashes(source)[1] def compatible_identity(source_hash: str, device: int | None, capacity: int) -> dict[str, object]: @@ -141,18 +135,51 @@ def _validate_shards(result: ShardedReplayResult) -> None: ) +def _compact_compatible_block_ids(result: ShardedReplayResult) -> None: + """Match Insight's continuous preexisting-block IDs without changing native shards.""" + for device in sorted({item.device for item in result.slices}): + items = [item for item in result.slices if item.device == device] + tokens: set[int] = set() + for item in items: + with closing(sqlite3.connect(item.file.as_uri() + "?mode=ro", uri=True)) as conn: + tokens.update( + row[0] for row in conn.execute(f"SELECT id FROM block_{device} WHERE id<0") + ) + if not tokens: + continue + ordered = sorted(tokens, reverse=True) + floor = min(tokens) - len(tokens) + # Move every key out of the final range before assigning compact IDs. + phases = ( + [(floor - index, token) for index, token in enumerate(ordered, 1)], + [(-index, floor - index) for index in range(1, len(ordered) + 1)], + ) + for item in items: + with closing(sqlite3.connect(item.file.as_uri() + "?mode=rw", uri=True)) as conn, conn: + for updates in phases: + conn.executemany(f"UPDATE block_{device} SET id=? WHERE id=?", updates) + conn.executemany( + "UPDATE pt_snap_block_reference SET blockId=? WHERE blockId=?", updates + ) + + def convert_shards(result: ShardedReplayResult) -> None: """Preserve event/stream/lifecycle values, materialize inline ninth column. First validate all selected IDs/actions, without inventing event zero or - coercing unsupported OOM. Block/reference tables are left byte-semantically - intact; no foreign real events are copied into the slice window. + coercing unsupported OOM. Negative block IDs and boundary IDs are normalized + to Insight numbering; no foreign real events are copied into the slice window. """ _validate_shards(result) - for item in result.slices: + _compact_compatible_block_ids(result) + boundary_count = 0 + for item in sorted(result.slices, key=lambda item: (item.device, -item.index)): device = item.device trace = f"trace_entry_{device}" with closing(sqlite3.connect(item.file.as_uri() + "?mode=rw", uri=True)) as conn, conn: + count = conn.execute(f"SELECT COUNT(*) FROM {trace} WHERE id<0").fetchone()[0] + conn.execute(f"UPDATE {trace} SET id=id-? WHERE id<0", (boundary_count,)) + boundary_count += count # A missing stack reference must fail, not silently erase an event. if ( conn.execute( @@ -210,9 +237,7 @@ def build_compatible_manifest( for s in slices ], } - for item in slices: - files[item.file.relative_to(result.directory).as_posix()] = hash_file(item.file) - return { + manifest: dict[str, Any] = { "schemaVersion": MANIFEST_SCHEMA_VERSION, "status": "complete", "sourceFile": str(source.resolve()), @@ -228,6 +253,10 @@ def build_compatible_manifest( "members": files, }, } + seal_manifest(result.directory, manifest, metadata.requested_device) + manifest["ptSnap"]["members"] = manifest["parserCache"]["members"] + manifest["ptSnap"]["memberHashKind"] = "parser-v1" + return manifest @dataclass(frozen=True) @@ -241,18 +270,34 @@ class CompatibleCache: def inspect_owned_cache(target: Path) -> CompatibleCache: """Whole valid recognized identity only; salted hash alone is never ownership. - Reject unexpected files/aliases rather than adopt them. GUI may change - derived tables: that invalidates pt-snap's immutable cache attestation, and - requires a new output, never repair or force-replacement. + Base tables are attested logically. Consumer-owned tables and indexes may + change without invalidating parser data. Legacy exports retain byte checks. """ - validation = validate_dataset(target) - with (target / "manifest.json").open(encoding="utf-8") as file: - data = json.load(file, object_pairs_hook=_unique_object) + data = parse_cache_manifest((target / "manifest.json").read_bytes()) + extension = data.get("ptSnap") + if not isinstance(extension, dict): + raise ValueError("manifest.ptSnap: expected object") + identity = extension.get("identity") + if not isinstance(identity, dict): + raise ValueError("manifest.ptSnap.identity: expected object") + for key in ("metadata", "members"): + if not isinstance(extension.get(key), dict): + raise ValueError(f"manifest.ptSnap.{key}: expected object") + if not isinstance(extension.get("omittedDevices"), list): + raise ValueError("manifest.ptSnap.omittedDevices: expected array") + version = identity.get("exportContractVersion") + if type(version) is int and version < EXPORT_VERSION: + raise ValueError( + f"Legacy export contract v{version}; choose a new --output-dir to re-export as v{EXPORT_VERSION}" + ) + # Logical seals establish consistency, not semantic validity. Keep shared P0 + # checks for every producer, including imports that do not register focus. + manifest = validate_dataset(target).manifest extension = data["ptSnap"] metadata = metadata_from_mapping(extension["metadata"]) identity = extension["identity"] expected_identity = compatible_identity( - metadata.source_sha256, metadata.requested_device, validation.manifest.events_per_slice + metadata.source_sha256, metadata.requested_device, manifest.events_per_slice ) if identity != expected_identity or any( type(identity[key]) is not type(value) for key, value in expected_identity.items() @@ -264,19 +309,42 @@ def inspect_owned_cache(target: Path) -> CompatibleCache: or metadata.metadata_schema_version != 1 ): raise ValueError("Metadata must match physical v1 layout") - paths = {s.file for d in validation.manifest.devices for s in d.slices} + paths = {s.file for d in manifest.devices for s in d.slices} + if any((target / (relative + "-journal")).exists() for relative in paths): + raise ValueError( + "Snapshot member has a rollback journal; a writer or recovery may be active, retry later" + ) if set(extension["members"]) != paths: raise ValueError("Incomplete member attestation") - expected = {"manifest.json", *paths, *(str(Path(p).parent) for p in paths)} + expected = {"manifest.json", "parse.log", *paths, *(str(Path(p).parent) for p in paths)} + expected.update(relative + suffix for relative in paths for suffix in ("-wal", "-shm")) actual = {p.relative_to(target).as_posix() for p in target.rglob("*")} - if actual != expected or any(p.is_symlink() for p in target.rglob("*")): + if not actual <= expected or any(p.is_symlink() for p in target.rglob("*")): raise ValueError("Unrecognized files or aliases in compatible cache") + logical_members = None + if extension.get("memberHashKind") == "parser-v1": + checked = validate_cache(target, data["cacheHash"], metadata.requested_device) + if checked != data: + raise ValueError("Compatible manifest changed during inspection") + # validate_cache checks the producer seal even without parserCache. + logical_members = checked["ptSnap"]["members"] + elif "memberHashKind" in extension: + raise ValueError("Unsupported member hash contract") for relative in paths: member = target / relative - if hash_file(member) != extension["members"][relative]: + digest = logical_members[relative] if logical_members is not None else hash_file(member) + if digest != extension["members"][relative]: raise ValueError("Compatible member hash changed") - with closing(sqlite3.connect(member.as_uri() + "?mode=ro&immutable=1", uri=True)) as conn: + require_readonly_member(target, member, immutable=True) + with closing(sqlite3.connect(readonly_member_uri(member), uri=True)) as conn: conn.row_factory = sqlite3.Row + if ( + conn.execute( + "SELECT 1 FROM sqlite_master WHERE type='table' AND name='pt_snap_metadata'" + ).fetchone() + is None + ): + raise ValueError(f"{relative}.pt_snap_metadata: required metadata table is missing") rows = conn.execute("SELECT * FROM pt_snap_metadata").fetchall() if ( len(rows) != 1 @@ -285,7 +353,7 @@ def inspect_owned_cache(target: Path) -> CompatibleCache: ): raise ValueError("Member metadata disagrees with identity") omitted = extension["omittedDevices"] - devices = {d.device_id for d in validation.manifest.devices} + devices = {d.device_id for d in manifest.devices} if ( not isinstance(omitted, list) or any(type(v) is not int or v < 0 or v in devices for v in omitted) @@ -294,6 +362,6 @@ def inspect_owned_cache(target: Path) -> CompatibleCache: raise ValueError("Invalid omitted device inventory") if metadata.requested_device is not None and devices != {metadata.requested_device}: raise ValueError("Device selection disagrees with identity") - if Path(validation.manifest.source_file).name != metadata.source_name: + if Path(manifest.source_file).name != metadata.source_name: raise ValueError("Source path disagrees with metadata") - return CompatibleCache(validation.manifest, identity, metadata, tuple(omitted)) + return CompatibleCache(manifest, identity, metadata, tuple(omitted)) diff --git a/src/pt_snap_cli/core/snapshot_cache.py b/src/pt_snap_cli/core/snapshot_cache.py new file mode 100644 index 0000000..5f23341 --- /dev/null +++ b/src/pt_snap_cli/core/snapshot_cache.py @@ -0,0 +1,341 @@ +"""Portable msinsight cache contract; consumer-owned tables are not parser data.""" + +from __future__ import annotations + +import hashlib +import json +import os +import re +import sqlite3 +import sys +from collections import OrderedDict +from contextlib import closing +from pathlib import Path +from typing import Any + +from .validation_budget import ( + ValidationBudget, + check_budget, + checked_rows, + sqlite_timeout, + validation_progress, +) + +PARSER_HASH_SALT = b"mem_snapshot_parser_v2" +CACHE_CONTRACT_VERSION = 1 +TRACE_COLUMNS = ( + "id", + "action", + "address", + "size", + "stream", + "allocated", + "active", + "reserved", + "callstack", +) +BLOCK_COLUMNS = ("id", "address", "size", "requestedSize", "state", "allocEventId", "freeEventId") + + +class _RowEncoder: + """Bound repeated callstack encoding; preserve the v1 digest bytes.""" + + def __init__(self) -> None: + self.stacks: OrderedDict[str, bytes] = OrderedDict() + self.size = 0 + + def encode(self, row: tuple[Any, ...], inline_stack: bool) -> bytes: + if not inline_stack or not isinstance(row[-1], str): + return json.dumps( + row, ensure_ascii=True, separators=(",", ":"), allow_nan=False + ).encode() + stack = row[-1] + encoded = self.stacks.get(stack) + if encoded is None: + encoded = json.dumps(stack, ensure_ascii=True, allow_nan=False).encode() + cost = sys.getsizeof(stack) + sys.getsizeof(encoded) + if cost <= 8 * 1024 * 1024: + while self.stacks and ( + self.size + cost > 8 * 1024 * 1024 or len(self.stacks) >= 2048 + ): + key, value = self.stacks.popitem(last=False) + self.size -= sys.getsizeof(key) + sys.getsizeof(value) + self.stacks[stack] = encoded + self.size += cost + else: + self.stacks.move_to_end(stack) + prefix = json.dumps(row[:-1], separators=(",", ":"), allow_nan=False).encode() + return prefix[:-1] + b"," + encoded + b"]" + + +def source_hashes(source: Path) -> tuple[str, str]: + """Return raw and salted SHA-256 in one streaming pass, without unpickling.""" + raw = hashlib.sha256() + salted = hashlib.sha256(PARSER_HASH_SALT) + with source.open("rb") as file: + before = os.fstat(file.fileno()) + for chunk in iter(lambda: file.read(1024 * 1024), b""): + raw.update(chunk) + salted.update(chunk) + after = os.fstat(file.fileno()) + current = source.stat() + + def signature(stat: os.stat_result) -> tuple[int, int, int, int]: + # Windows fstat/stat disagree on ctime for some files; it is not content identity. + return stat.st_dev, stat.st_ino, stat.st_size, stat.st_mtime_ns + + if signature(before) != signature(after) or signature(after) != signature(current): + raise ValueError("Snapshot source changed while hashing") + return raw.hexdigest(), salted.hexdigest() + + +def _integer(value: Any, minimum: int = 0) -> int: + if ( + not isinstance(value, int) + or isinstance(value, bool) + or value < minimum + or value > 2**63 - 1 + ): + raise ValueError("Invalid nonnegative integer in snapshot manifest") + return value + + +def _member(directory: Path, relative: str) -> Path: + if not isinstance(relative, str) or not relative or "\\" in relative: + raise ValueError("Invalid snapshot member path") + path = Path(relative) + if path.is_absolute() or any(part in (".", "..") for part in relative.split("/")): + raise ValueError("Snapshot member escapes artifact directory") + candidate = directory / path + if not candidate.is_file() or candidate.resolve() != directory.resolve() / path: + raise ValueError("Missing or aliased snapshot member") + return candidate + + +def _table_rows( + conn: sqlite3.Connection, + name: str, + columns: tuple[str, ...], + where: str = "", + parameters: tuple[str, ...] = (), +) -> sqlite3.Cursor: + if conn.execute("SELECT type FROM sqlite_master WHERE name=?", (name,)).fetchone() != ( + "table", + ): + raise ValueError(f"Missing parser table: {name}") + schema = conn.execute(f'PRAGMA table_info("{name}")').fetchall() + actual = {row[1]: row for row in schema} + for column in columns: + kind = "TEXT" if column in ("callstack", "table", "column", "key", "value") else "INTEGER" + if column not in actual or actual[column][2].upper() != kind: + raise ValueError(f"Invalid parser column: {name}.{column}") + if "id" in columns and [row[1] for row in schema if row[5]] != ["id"]: + raise ValueError(f"Missing parser primary key: {name}.id") + selected = ",".join(f'"{column}"' for column in columns) + order = '"id"' if "id" in columns else selected + # Identifiers and the optional predicate are internal constants/validated device IDs. + return conn.execute( + f'SELECT {selected} FROM "{name}" {where} ORDER BY {order}', parameters + ) # nosec B608 + + +def member_digest( + path: Path, device: str, start: int, end: int, *, budget: ValidationBudget | None = None +) -> str: + """Hash fixed base columns; consumer columns, tables, indexes, views and triggers are allowed.""" + if re.fullmatch(r"0|[1-9][0-9]*", str(device)) is None: + raise ValueError("Invalid device ID") + digest = hashlib.sha256() + encoder = _RowEncoder() + # Consumers may start writing after the sidecar check. Always use SQLite's + # locking/WAL protocol and one read snapshot, even for a currently closed DB. + sidecars = [Path(str(path) + suffix) for suffix in ("-wal", "-shm", "-journal")] + if any(item.is_symlink() for item in sidecars): + raise ValueError("Snapshot has an aliased sidecar") + if sidecars[2].exists(): + raise ValueError( + "Snapshot has a rollback journal; a writer or recovery may be active, retry later" + ) + uri = path.resolve().as_uri() + "?mode=ro" + with ( + closing(sqlite3.connect(uri, uri=True, timeout=sqlite_timeout(budget))) as conn, + validation_progress(conn, budget), + ): + conn.execute("PRAGMA query_only=ON") + conn.execute("BEGIN") + if conn.execute("PRAGMA quick_check").fetchall() != [("ok",)]: + raise ValueError("Snapshot member failed integrity check") + tables = ((f"trace_entry_{device}", TRACE_COLUMNS), (f"block_{device}", BLOCK_COLUMNS)) + for table, columns in tables: + digest.update(json.dumps((table, columns), separators=(",", ":")).encode()) + for row in checked_rows(_table_rows(conn, table, columns), budget): + digest.update(encoder.encode(row, columns == TRACE_COLUMNS)) + digest.update(b"\n") + count, low, high = conn.execute( + f'SELECT COUNT(*), MIN(id), MAX(id) FROM "trace_entry_{device}" WHERE id>=0' # nosec B608 - device validated above + ).fetchone() + if (count, low, high) != (end - start + 1, start, end): + raise ValueError("Snapshot event range is incomplete") + for row in _table_rows( + conn, + "dictionary", + ("table", "column", "key", "value"), + 'WHERE "table" IN (?, ?)', + tuple(table for table, _ in tables), + ): + digest.update(json.dumps(row, ensure_ascii=True, separators=(",", ":")).encode()) + digest.update(b"\n") + for table, column in ((f"trace_entry_{device}", "action"), (f"block_{device}", "state")): + # Identifiers are internal constants with the device validated above. + missing = conn.execute( + f'SELECT 1 FROM (SELECT DISTINCT "{column}" AS value FROM "{table}") AS base WHERE NOT EXISTS ' # nosec B608 + '(SELECT 1 FROM dictionary AS dict WHERE dict."table"=? AND dict."column"=? ' + 'AND dict."key"=CAST(base.value AS TEXT)) LIMIT 1', + (table, column), + ).fetchone() + if missing: + raise ValueError("Snapshot dictionary reference is missing") + return digest.hexdigest() + + +def parser_members( + directory: Path, manifest: dict[str, Any], *, budget: ValidationBudget | None = None +) -> dict[str, str]: + """Validate the complete manifest and all DBs; never deserialize the source.""" + if _integer(manifest.get("schemaVersion"), 1) != 1: + raise ValueError("Unsupported snapshot manifest version") + if ( + manifest.get("status") != "complete" + or not isinstance(manifest.get("sourceFile"), str) + or not manifest["sourceFile"] + ): + raise ValueError("Snapshot is incomplete") + capacity = _integer(manifest.get("eventsPerSlice"), 1) + devices = manifest.get("devices") + if not isinstance(devices, dict) or not devices: + raise ValueError("Missing snapshot devices") + members = {} + for device, data in devices.items(): + if re.fullmatch(r"0|[1-9][0-9]*", device) is None: + raise ValueError("Invalid device ID") + if not isinstance(data, dict): + raise ValueError("Invalid device inventory") + count = _integer(data["eventCount"], 1) + size = _integer(data["sliceCount"], 1) + if size != (count + capacity - 1) // capacity or len(data["slices"]) != size: + raise ValueError("Incomplete snapshot slice inventory") + if sorted(_integer(index) for index in data["readySlices"]) != list(range(size)): + raise ValueError("Snapshot slices are not ready") + for index, item in enumerate(data["slices"]): + check_budget(budget) + if not isinstance(item, dict): + raise ValueError(f"manifest.devices.{device}.slices[{index}]: expected object") + start, end = index * capacity, min((index + 1) * capacity, count) - 1 + if ( + _integer(item["index"]), + _integer(item["startEventId"]), + _integer(item["endEventId"]), + ) != (index, start, end) or item["ready"] is not True: + raise ValueError("Invalid snapshot slice range") + relative = item["file"] + if relative in members: + raise ValueError("Duplicate snapshot member") + members[relative] = member_digest( + _member(directory, relative), device, start, end, budget=budget + ) + return members + + +def seal_manifest(directory: Path, manifest: dict[str, Any], device: int | None = None) -> None: + manifest["parserCache"] = { + "version": CACHE_CONTRACT_VERSION, + "requestedDevice": device, + "members": parser_members(directory, manifest), + } + + +def _unique_object(pairs: list[tuple[str, Any]]) -> dict[str, Any]: + result = {} + for key, value in pairs: + if key in result: + raise ValueError("Duplicate snapshot manifest key") + result[key] = value + return result + + +def parse_cache_manifest(raw: bytes) -> dict[str, Any]: + """Normalize malformed/deep JSON consistently at every compatible import entry.""" + try: + manifest = json.loads(raw, object_pairs_hook=_unique_object) + except RecursionError as error: + raise ValueError("Snapshot manifest nesting is too deep") from error + if not isinstance(manifest, dict): + raise ValueError("Invalid snapshot manifest") + return manifest + + +def validate_cache( + directory: Path, + cache_hash: str, + device: int | None = None, + *, + budget: ValidationBudget | None = None, +) -> dict[str, Any]: + path = directory / "manifest.json" + original = path.read_bytes() + manifest = parse_cache_manifest(original) + if not cache_hash or manifest.get("cacheHash") != cache_hash: + raise ValueError("Snapshot source hash or parser salt changed") + if device is not None and set(manifest["devices"]) != {str(device)}: + raise ValueError("Snapshot device selection changed") + for extension in ("parserCache", "ptSnap"): + if extension in manifest and not isinstance(manifest[extension], dict): + raise ValueError("Invalid snapshot extension") + if not isinstance(manifest.get("ptSnap", {}).get("identity", {}), dict): + raise ValueError("Invalid producer identity") + # Validate each declaration independently; an absent selection in one seal + # must not mask a producer's device-filtered export. + for selection in ( + manifest.get("parserCache", {}), + manifest.get("ptSnap", {}).get("identity", {}), + ): + selected_device = selection.get("requestedDevice") + if selected_device is None: + continue + _integer(selected_device) + if device is None: + raise ValueError("Snapshot cache contains only a selected device") + if selected_device != device or set(manifest["devices"]) != {str(selected_device)}: + raise ValueError("Snapshot device selection changed") + members = parser_members(directory, manifest, budget=budget) + attestation = manifest.get("parserCache") + # Legacy producers have no logical attestation; still validate every base table and range. + if attestation is not None and ( + _integer(attestation.get("version"), 1) != CACHE_CONTRACT_VERSION + or attestation.get("members") != members + ): + raise ValueError("Snapshot parser data changed") + # Both attestations must hold; parserCache must not hide an older producer seal. + producer = manifest.get("ptSnap", {}) + if "memberHashKind" in producer and producer["memberHashKind"] != "parser-v1": + raise ValueError("Unsupported member hash contract") + if producer.get("memberHashKind") == "parser-v1" and producer.get("members") != members: + raise ValueError("Snapshot producer member hash changed") + if "members" in producer and producer.get("memberHashKind") != "parser-v1": + physical = producer["members"] + if not isinstance(physical, dict) or set(physical) != set(members): + raise ValueError("Invalid legacy snapshot member inventory") + from .dataset_files import hash_file + + for relative, expected in physical.items(): + member = _member(directory, relative) + wal = Path(str(member) + "-wal") + # A read-only open can create an empty WAL and SHM. They contain no + # committed rows; only the DB bytes need the legacy physical digest. + if Path(str(member) + "-journal").exists() or (wal.exists() and wal.stat().st_size): + raise ValueError("Legacy physical snapshot hash cannot validate live sidecars") + if hash_file(member, budget=budget) != expected: + raise ValueError("Legacy snapshot member hash changed") + if path.read_bytes() != original: + raise ValueError("Snapshot manifest changed during validation") + return manifest diff --git a/tests/core/test_dataset_validation_cache.py b/tests/core/test_dataset_validation_cache.py index b5ef817..8a356ef 100644 --- a/tests/core/test_dataset_validation_cache.py +++ b/tests/core/test_dataset_validation_cache.py @@ -32,6 +32,65 @@ def artifact(tmp_path, request): return native_dataset(tmp_path) if request.param else make_dataset(tmp_path / "dataset") +def test_closed_wal_compatibility_dataset_opens_on_first_inspection(tmp_path, monkeypatch): + monkeypatch.delenv("PT_SNAP_DATASET_CACHE", raising=False) + root = make_dataset(tmp_path / "dataset") + for member in root.glob("*/*.db"): + with closing(sqlite3.connect(member)) as conn: + conn.execute("PRAGMA journal_mode=WAL") + assert not Path(str(member) + "-wal").exists() + resolved = DatasetResolver().inspect(root) + assert resolved is not None + resolved.require_unchanged() + assert DatasetResolver().inspect(root).fingerprint == resolved.fingerprint + + +def test_wal_preparation_does_not_hide_subsequent_content_changes(tmp_path, monkeypatch): + monkeypatch.delenv("PT_SNAP_DATASET_CACHE", raising=False) + root = make_dataset(tmp_path / "dataset", devices=(0,), slices=1) + member = root / "device_0/slice_00000.db" + with closing(sqlite3.connect(member)) as conn: + conn.execute("PRAGMA journal_mode=WAL") + validate = dataset_resolver.validate_dataset + + def changed(*args, **kwargs): + result = validate(*args, **kwargs) + with closing(sqlite3.connect(member)) as writer, writer: + writer.execute("UPDATE trace_entry_0 SET callstack='changed after validation'") + return result + + monkeypatch.setattr(dataset_resolver, "validate_dataset", changed) + with pytest.raises(DatabaseSchemaError, match="changed during inspection"): + DatasetResolver().inspect(root) + + +@pytest.mark.parametrize("cache_mode", [None, "immutable", "disabled"]) +@pytest.mark.parametrize("seal", ["parserCache", "ptSnap"]) +def test_declared_seals_are_checked_in_every_cache_mode(tmp_path, monkeypatch, cache_mode, seal): + from pt_snap_cli.core.snapshot_cache import seal_manifest + + if cache_mode is None: + monkeypatch.delenv("PT_SNAP_DATASET_CACHE", raising=False) + else: + monkeypatch.setenv("PT_SNAP_DATASET_CACHE", cache_mode) + root = make_dataset(tmp_path / "dataset", devices=(0,), slices=1) + manifest_path = root / "manifest.json" + manifest = json.loads(manifest_path.read_bytes()) + manifest["cacheHash"] = "a" * 64 + seal_manifest(root, manifest) + if seal == "ptSnap": + manifest["ptSnap"] = { + "memberHashKind": "parser-v1", + "members": manifest.pop("parserCache")["members"], + } + manifest_path.write_text(json.dumps(manifest)) + assert DatasetResolver().inspect(root) is not None + with closing(sqlite3.connect(root / "device_0/slice_00000.db")) as conn, conn: + conn.execute("UPDATE trace_entry_0 SET allocated=allocated+1") + with pytest.raises(DatabaseSchemaError, match="seal"): + DatasetResolver().inspect(root) + + def forbidden(*args, **kwargs): pytest.fail("warm resolution must not hash or validate shard contents") diff --git a/tests/core/test_msinsight_export.py b/tests/core/test_msinsight_export.py index 4cd810c..ef2b5e8 100644 --- a/tests/core/test_msinsight_export.py +++ b/tests/core/test_msinsight_export.py @@ -108,7 +108,12 @@ def test_physical_schema_inline_events_streams_stable_references_and_native_pari native.db_path / native_shard.file, f"SELECT t.id,t.action,t.address,t.size,t.stream,t.allocated,t.active,t.reserved,c.callstack FROM trace_entry_{device} t JOIN callstack c ON c.id=t.callstackId ORDER BY t.id", ) - assert actual == expected + assert [row for row in actual if row[0] >= 0] == [ + row for row in expected if row[0] >= 0 + ] + assert sorted(row[1:] for row in actual if row[0] < 0) == sorted( + row[1:] for row in expected if row[0] < 0 + ) assert [r[0] for r in actual if r[0] >= 0] == list( range(shard.start_event_id, shard.end_event_id + 1) ) @@ -161,6 +166,450 @@ def test_identical_complete_cache_reuses_without_loading_even_force(tmp_path, mo assert not (tmp_path / ".pt-snap").exists() +def test_consumer_tables_do_not_invalidate_compatible_cache(tmp_path, monkeypatch): + request = options(tmp_path, focus=False) + service = ImportService() + first = service.import_snapshot(request) + member = first.db_path / "device_0/slice_00000.db" + with closing(sqlite3.connect(member)) as conn, conn: + conn.executescript( + "CREATE TABLE memory_allocation_cache_v1_0(value TEXT);" + "CREATE TABLE unrelated_consumer_cache(value TEXT);" + "INSERT INTO unrelated_consumer_cache VALUES('keep');" + "CREATE INDEX consumer_block_index ON block_0(address);" + "ALTER TABLE trace_entry_0 ADD COLUMN consumer_note TEXT;" + "ALTER TABLE block_0 ADD COLUMN consumer_note TEXT;" + ) + (first.db_path / "parse.log").write_text("Insight log") + monkeypatch.setattr( + service._msinsight_backend.replay, + "stage", + lambda *a, **k: pytest.fail("cache hit must not replay"), + ) + assert service.import_snapshot(request).reused + with SnapshotAnalyzer(first.db_path) as analyzer: + assert analyzer.execute_query("event", {"id": 0}, device_id=0)["rows"] + assert analyzer.execute_query("block", {}, device_id=0)["rows"] + assert analyzer.get_database_metadata()["status"] == "available" + assert rows(member, "SELECT value FROM unrelated_consumer_cache") == [("keep",)] + with closing(sqlite3.connect(member)) as conn, conn: + conn.execute("UPDATE block_0 SET size=size+1") + with pytest.raises(ImportExecutionError, match="preserved"): + service.import_snapshot(request) + + +def test_producer_only_logical_seal_reuses_without_replaying(tmp_path, monkeypatch): + request = options(tmp_path, focus=False) + service = ImportService() + first = service.import_snapshot(request) + path = first.db_path / "manifest.json" + manifest = json.loads(path.read_text()) + del manifest["parserCache"] + path.write_text(json.dumps(manifest)) + before = hashes(first.db_path) + monkeypatch.setattr( + service._msinsight_backend.replay, + "stage", + lambda *a, **k: pytest.fail("producer seal cache hit must not replay"), + ) + reused = service.import_snapshot(request) + assert reused.reused and reused.metadata == first.metadata + assert hashes(first.db_path) == before + with SnapshotAnalyzer(first.db_path) as analyzer: + assert analyzer.get_database_metadata()["status"] == "available" + with closing(sqlite3.connect(first.db_path / "device_0/slice_00000.db")) as conn, conn: + conn.execute("UPDATE block_0 SET size=size+1") + with pytest.raises(ImportExecutionError, match="preserved"): + service.import_snapshot(request) + + +@pytest.mark.parametrize("cli_capacity", [None, 2]) +def test_external_insight_artifact_reuses_without_claiming_metadata( + tmp_path, monkeypatch, cli_capacity +): + request = options(tmp_path, focus=False) + service = ImportService() + first = service.import_snapshot(request) + path = first.db_path / "manifest.json" + manifest = json.loads(path.read_text()) + del manifest["ptSnap"] + path.write_text(json.dumps(manifest)) + before = hashes(first.db_path) + monkeypatch.setattr( + service._msinsight_backend.replay, + "stage", + lambda *a, **k: pytest.fail("external cache hit must not replay"), + ) + reused = service.import_snapshot(request) + assert reused.reused and reused.metadata is None + assert hashes(first.db_path) == before + args = [ + "import", + str(request.snapshot_file), + "--format", + "msinsight", + "--output-dir", + str(request.output_dir), + "--no-focus", + "--json", + ] + if cli_capacity is not None: + args.extend(["--events-per-slice", str(cli_capacity)]) + output = CliRunner().invoke(app, args) + assert output.exit_code == 0, output.output + payload = json.loads(output.stdout) + assert payload["metadata"] is None and payload["reused"] and payload["slice_count"] == 3 + with SnapshotAnalyzer(first.db_path) as analyzer: + assert analyzer.execute_query("event", {"id": 4}, device_id=0)["rows"] + assert analyzer.execute_query("block", {}, device_id=0)["rows"] + assert hashes(first.db_path) == before + + +@pytest.mark.parametrize("sealed", [False, True]) +@pytest.mark.parametrize("external", [False, True]) +@pytest.mark.parametrize("set_focus", [False, True]) +@pytest.mark.parametrize( + "slice_index,mutation,diagnostic", + [ + (0, "UPDATE trace_entry_0 SET allocated=-1 WHERE id=0", "event_values"), + (0, "UPDATE trace_entry_0 SET action=4 WHERE id<0", "event_action"), + (0, "UPDATE block_0 SET requestedSize=-1", "block: invalid state/size"), + (0, "UPDATE block_0 SET freeEventId=-2", "lifecycle"), + (1, "UPDATE block_0 SET address=address+1 WHERE id=0", "block_identity"), + (2, "UPDATE trace_entry_0 SET reserved=-1 WHERE id=4", "event_values"), + ], + ids=[ + "negative-bytes", + "boundary-action", + "block-size", + "lifetime", + "cross-slice", + "last-slice", + ], +) +def test_compatible_cache_requires_semantic_validation( + tmp_path, monkeypatch, sealed, external, set_focus, slice_index, mutation, diagnostic +): + from pt_snap_cli.core.errors import DatabaseSchemaError + from pt_snap_cli.core.snapshot_cache import seal_manifest, validate_cache + + request = options(tmp_path, focus=False) + service = ImportService() + result = service.import_snapshot(request) + path = result.db_path / "manifest.json" + manifest = json.loads(path.read_text()) + if external: + del manifest["ptSnap"] + member = result.db_path / f"device_0/slice_{slice_index:05d}.db" + with closing(sqlite3.connect(member)) as conn, conn: + assert conn.execute(mutation).rowcount > 0 + seal_manifest(result.db_path, manifest) + if not external: + manifest["ptSnap"]["members"] = manifest["parserCache"]["members"] + if not sealed: + del manifest["parserCache"] + path.write_text(json.dumps(manifest)) + # The digest is self-consistent, but must not replace shared P0 validation. + validate_cache(result.db_path, manifest["cacheHash"]) + with pytest.raises(DatabaseSchemaError, match=diagnostic): + DatasetResolver().inspect(result.db_path) + before = hashes(result.db_path) + source_before = request.snapshot_file.read_bytes() + focus = tmp_path / ".pt-snap/focus.json" + focus.parent.mkdir() + focus.write_bytes(b'{"db_path":"/previous/project.db","device_id":0}') + focus_before = focus.read_bytes() + monkeypatch.setattr( + service._msinsight_backend.replay, "stage", lambda *a, **k: pytest.fail("must preserve") + ) + for force in (False, True): + with pytest.raises(ImportExecutionError, match=diagnostic): + service.import_snapshot(dataclasses.replace(request, force=force, set_focus=set_focus)) + assert hashes(result.db_path) == before + assert request.snapshot_file.read_bytes() == source_before + assert focus.read_bytes() == focus_before + + +@pytest.mark.parametrize("capacity", [None, 2, 3]) +@pytest.mark.parametrize("force", [False, True]) +@pytest.mark.parametrize("sealed", [False, True]) +def test_external_cache_honors_explicit_capacity(tmp_path, monkeypatch, capacity, force, sealed): + request = options(tmp_path, focus=False) + service = ImportService() + result = service.import_snapshot(request) + path = result.db_path / "manifest.json" + manifest = json.loads(path.read_text()) + del manifest["ptSnap"] + if not sealed: + del manifest["parserCache"] + path.write_text(json.dumps(manifest)) + before = hashes(result.db_path) + monkeypatch.setattr( + service._msinsight_backend.replay, "stage", lambda *a, **k: pytest.fail("must not replay") + ) + updated = dataclasses.replace(request, events_per_slice=capacity, force=force) + if capacity == 3: + with pytest.raises(ImportExecutionError, match="events_per_slice.*new output directory"): + service.import_snapshot(updated) + else: + reused = service.import_snapshot(updated) + assert reused.reused and reused.metadata is None and reused.slice_count == 3 + assert hashes(result.db_path) == before + assert not (tmp_path / ".pt-snap").exists() + + +@pytest.mark.parametrize("force", [False, True]) +@pytest.mark.parametrize("failure", ["semantic", "capacity"]) +def test_external_cache_cli_failure_preserves_artifact_and_focus( + tmp_path, monkeypatch, force, failure +): + from pt_snap_cli.core.snapshot_cache import seal_manifest + + request = options(tmp_path, focus=False) + result = ImportService().import_snapshot(request) + path = result.db_path / "manifest.json" + manifest = json.loads(path.read_text()) + del manifest["ptSnap"] + if failure == "semantic": + with closing(sqlite3.connect(result.db_path / "device_0/slice_00000.db")) as conn, conn: + conn.execute("UPDATE trace_entry_0 SET allocated=-1 WHERE id=0") + seal_manifest(result.db_path, manifest) + path.write_text(json.dumps(manifest)) + before = hashes(result.db_path) + source_before = request.snapshot_file.read_bytes() + focus = tmp_path / ".pt-snap/focus.json" + focus.parent.mkdir() + focus.write_bytes(b'{"db_path":"/previous/project.db","device_id":0}') + focus_before = focus.read_bytes() + monkeypatch.setattr( + "pt_snap_cli.core.msinsight_export.MsinsightImportBackend.dump_to_dataset", + lambda *a, **k: pytest.fail("rejected external cache must not replay or publish"), + ) + args = [ + "import", + str(request.snapshot_file), + "--format", + "msinsight", + "--output-dir", + str(request.output_dir), + "--json", + ] + if failure == "capacity": + args.extend(["--events-per-slice", "3"]) + if force: + args.append("--force") + response = CliRunner().invoke(app, args) + assert response.exit_code == 1, response.output + assert response.stdout == "" + error = json.loads(response.stderr) + assert error["ok"] is False and error["error"]["code"] == "IMPORT_FAILED" + diagnostic = "event_values" if failure == "semantic" else "events_per_slice" + assert diagnostic in error["error"]["message"] + assert "Traceback" not in response.stderr + assert hashes(result.db_path) == before + assert request.snapshot_file.read_bytes() == source_before + assert focus.read_bytes() == focus_before + + +@pytest.mark.parametrize("external", [False, True]) +@pytest.mark.parametrize("failure", ["index", "dictionary", "deep-json"]) +def test_invalid_compatible_cache_is_preserved(tmp_path, monkeypatch, external, failure): + request = options(tmp_path, focus=False) + service = ImportService() + result = service.import_snapshot(request) + path = result.db_path / "manifest.json" + manifest = json.loads(path.read_text()) + if external: + del manifest["ptSnap"] + member = result.db_path / "device_0/slice_00000.db" + if failure == "index": + with closing(sqlite3.connect(member)) as conn: + conn.execute("CREATE INDEX block_address ON block_0(address)") + page_size = conn.execute("PRAGMA page_size").fetchone()[0] + root = conn.execute( + "SELECT rootpage FROM sqlite_master WHERE name='block_address'" + ).fetchone()[0] + with member.open("r+b") as file: + file.seek((root - 1) * page_size) + file.write(b"\xff" * page_size) + elif failure == "dictionary": + del manifest["parserCache"] + with closing(sqlite3.connect(member)) as conn, conn: + conn.execute("DELETE FROM dictionary WHERE \"table\"='block_0'") + path.write_text( + "[" * 200000 + "0" + "]" * 200000 if failure == "deep-json" else json.dumps(manifest) + ) + before = hashes(result.db_path) + monkeypatch.setattr( + service._msinsight_backend.replay, "stage", lambda *a, **k: pytest.fail("must preserve") + ) + for force in (False, True): + with pytest.raises(ImportExecutionError, match="preserved"): + service.import_snapshot(dataclasses.replace(request, force=force)) + assert hashes(result.db_path) == before + + +def test_owned_cache_inspection_normalizes_deep_json(tmp_path): + from pt_snap_cli.core.msinsight_export import inspect_owned_cache + + (tmp_path / "manifest.json").write_text("[" * 200000 + "0" + "]" * 200000) + with pytest.raises(ValueError, match="nesting"): + inspect_owned_cache(tmp_path) + + +def test_query_and_metadata_reject_deep_manifest_without_traceback(tmp_path): + from pt_snap_cli.core.errors import DatabaseSchemaError + from pt_snap_cli.core.import_metadata import ImportMetadataService + + result = ImportService().import_snapshot(options(tmp_path, focus=False)) + path = result.db_path / "manifest.json" + path.write_text("[" * 200000 + "0" + "]" * 200000) + before = hashes(result.db_path) + with pytest.raises(DatabaseSchemaError): + DatasetResolver().inspect(result.db_path) + with pytest.raises(DatabaseSchemaError): + ImportMetadataService().inspect(result.db_path) + for command in ("overview", "metadata"): + response = CliRunner().invoke(app, [command, str(result.db_path), "--json"]) + assert response.exit_code != 0 + assert "Traceback" not in response.output + assert hashes(result.db_path) == before + + +def test_rollback_journal_has_an_explicit_diagnostic(tmp_path): + request = options(tmp_path, focus=False) + result = ImportService().import_snapshot(request) + journal = result.db_path / "device_0/slice_00000.db-journal" + journal.write_bytes(b"") + before = hashes(result.db_path) + with pytest.raises(ImportExecutionError, match="rollback journal.*retry later"): + ImportService().import_snapshot(request) + assert hashes(result.db_path) == before + + +def test_selected_device_export_still_supports_sealed_queries(tmp_path): + request = dataclasses.replace(options(tmp_path, multiple=True, focus=False), device=1) + result = ImportService().import_snapshot(request) + assert result.devices == (1,) + with SnapshotAnalyzer(result.db_path) as analyzer: + assert analyzer.execute_query("event", {"id": 0}, device_id=1)["rows"] + + +@pytest.mark.parametrize( + ("field", "value", "message"), + [ + ("ptSnap", [], "manifest.ptSnap: expected object"), + ("identity", [], "manifest.ptSnap.identity: expected object"), + ("exportContractVersion", 1, "Legacy export contract v1"), + ], +) +def test_invalid_producer_extension_reports_context(tmp_path, field, value, message): + request = options(tmp_path, focus=False) + service = ImportService() + result = service.import_snapshot(request) + path = result.db_path / "manifest.json" + manifest = json.loads(path.read_text()) + if field == "ptSnap": + manifest[field] = value + elif field == "identity": + manifest["ptSnap"][field] = value + else: + manifest["ptSnap"]["identity"][field] = value + path.write_text(json.dumps(manifest)) + before = hashes(result.db_path) + with pytest.raises(ImportExecutionError, match=message): + service.import_snapshot(request) + assert hashes(result.db_path) == before + + +@pytest.mark.parametrize("external", [False, True]) +@pytest.mark.parametrize("mutation", ["block", "trace"]) +def test_resolver_rejects_seal_mismatch_after_prior_query(tmp_path, external, mutation): + result = ImportService().import_snapshot(options(tmp_path, focus=False)) + if external: + path = result.db_path / "manifest.json" + manifest = json.loads(path.read_text()) + del manifest["ptSnap"] + path.write_text(json.dumps(manifest)) + resolver = DatasetResolver() + assert resolver.inspect(result.db_path) is not None + member = result.db_path / "device_0/slice_00000.db" + with closing(sqlite3.connect(member)) as conn, conn: + conn.execute( + "DELETE FROM block_0 WHERE id=(SELECT MIN(id) FROM block_0)" + if mutation == "block" + else "UPDATE trace_entry_0 SET allocated=allocated+1" + ) + before = hashes(result.db_path) + from pt_snap_cli.core.errors import DatabaseSchemaError + + with pytest.raises(DatabaseSchemaError, match="parserCache.*seal"): + resolver.inspect(result.db_path) + assert hashes(result.db_path) == before + + +@pytest.mark.parametrize("external", [False, True]) +def test_live_consumer_wal_reuses_and_queries_committed_data(tmp_path, monkeypatch, external): + request = options(tmp_path, focus=False) + service = ImportService() + result = service.import_snapshot(request) + if external: + path = result.db_path / "manifest.json" + manifest = json.loads(path.read_text()) + del manifest["ptSnap"] + path.write_text(json.dumps(manifest)) + monkeypatch.setattr( + service._msinsight_backend.replay, "stage", lambda *a, **k: pytest.fail("must reuse") + ) + member = result.db_path / "device_0/slice_00000.db" + with closing(sqlite3.connect(member)) as writer: + writer.execute("PRAGMA journal_mode=WAL") + writer.executescript( + "CREATE TABLE consumer_data(value TEXT); INSERT INTO consumer_data VALUES('committed');" + ) + assert Path(str(member) + "-wal").exists() + reused = service.import_snapshot(request) + assert reused.reused and reused.metadata == (None if external else result.metadata) + with SnapshotAnalyzer(result.db_path) as analyzer: + assert analyzer.execute_query("event", {"id": 0}, device_id=0)["rows"] + assert writer.execute("SELECT value FROM consumer_data").fetchall() == [("committed",)] + # Base-table writes cannot masquerade as consumer-only additions. + writer.execute("UPDATE block_0 SET size=size+1") + writer.commit() + with pytest.raises(ImportExecutionError, match="preserved"): + service.import_snapshot(request) + + +def test_compatible_negative_ids_are_compact_and_boundary_ids_are_global(tmp_path): + result = ImportService().import_snapshot( + options(tmp_path, multiple=True, capacity=1, focus=False) + ) + manifest = json.loads((result.db_path / "manifest.json").read_text()) + boundary_ids = [] + for device, data in manifest["devices"].items(): + block_ids = set() + for item in reversed(data["slices"]): + path = result.db_path / item["file"] + block_ids.update( + row[0] for row in rows(path, f"SELECT id FROM block_{device} WHERE id<0") + ) + boundary_ids.extend( + row[0] + for row in rows( + path, f"SELECT id FROM trace_entry_{device} WHERE id<0 ORDER BY id DESC" + ) + ) + assert ( + rows( + path, + f"SELECT id FROM block_{device} EXCEPT SELECT blockId FROM pt_snap_block_reference", + ) + == [] + ) + assert block_ids == set(range(-len(block_ids), 0)) + assert boundary_ids == list(range(-1, -len(boundary_ids) - 1, -1)) + + @pytest.mark.parametrize( "axis", [ diff --git a/tests/core/test_msinsight_export_validation.py b/tests/core/test_msinsight_export_validation.py index b60b3e7..2d63f1b 100644 --- a/tests/core/test_msinsight_export_validation.py +++ b/tests/core/test_msinsight_export_validation.py @@ -39,6 +39,8 @@ def shard( ) conn.execute("CREATE TABLE callstack (id INTEGER PRIMARY KEY, callstack TEXT)") conn.execute("INSERT INTO callstack VALUES (0, 'synthetic stack')") + # Conversion also compacts preexisting block IDs, even in trace-only tests. + conn.execute(f"CREATE TABLE block_{device} (id INTEGER PRIMARY KEY)") conn.execute('CREATE TABLE dictionary ("table" TEXT, "column" TEXT, key TEXT, value TEXT)') real = [value for value in ids if value is not None and value >= 0] return ReplaySlice( diff --git a/tests/core/test_snapshot_cache.py b/tests/core/test_snapshot_cache.py new file mode 100644 index 0000000..0cf6d01 --- /dev/null +++ b/tests/core/test_snapshot_cache.py @@ -0,0 +1,419 @@ +"""Cache validation must not depend on consumer-added SQLite objects.""" + +import hashlib +import json +import sqlite3 +import tempfile +import unittest +from contextlib import closing +from pathlib import Path + +from pt_snap_cli.core.snapshot_cache import ( + _RowEncoder, + seal_manifest, + source_hashes, + validate_cache, +) + + +class SnapshotCacheTest(unittest.TestCase): + def test_row_encoder_preserves_digest_bytes_and_bounds_cache(self): + encoder = _RowEncoder() + for stack in ("", "ascii", "\u4e2d\u6587", '"\\\n\t', None, "x" * (8 * 1024 * 1024)): + row = (1, 2, 3, 4, 0, 5, 6, 7, stack) + expected = json.dumps( + row, ensure_ascii=True, separators=(",", ":"), allow_nan=False + ).encode() + self.assertEqual(encoder.encode(row, True), expected) + self.assertEqual(encoder.encode(row, True), expected) + for index in range(3000): + encoder.encode((index, str(index)), True) + self.assertLessEqual(len(encoder.stacks), 2048) + self.assertLessEqual(encoder.size, 8 * 1024 * 1024) + + def setUp(self): + self.temp = tempfile.TemporaryDirectory() + self.addCleanup(self.temp.cleanup) + self.directory = Path(self.temp.name) + self.source = self.directory / "input.pkl" + self.source.write_bytes(b"trusted-test-source") + self.hash = source_hashes(self.source)[1] + self.db = self.directory / "slice.db" + with closing(sqlite3.connect(self.db)) as conn, conn: + conn.executescript( + "CREATE TABLE trace_entry_0(id INTEGER PRIMARY KEY, action INTEGER, address INTEGER, " + "size INTEGER, stream INTEGER, allocated INTEGER, active INTEGER, reserved INTEGER, callstack TEXT);" + "CREATE TABLE block_0(id INTEGER PRIMARY KEY, address INTEGER, size INTEGER, requestedSize INTEGER, " + "state INTEGER, allocEventId INTEGER, freeEventId INTEGER);" + 'CREATE TABLE dictionary("table" TEXT, "column" TEXT, "key" TEXT, "value" TEXT);' + "INSERT INTO trace_entry_0 VALUES(0,4,100,10,0,10,10,20,'stack');" + "INSERT INTO block_0 VALUES(0,100,10,10,1,0,-1);" + "INSERT INTO dictionary VALUES('trace_entry_0','action','4','alloc');" + "INSERT INTO dictionary VALUES('block_0','state','1','active_allocated');" + ) + self.manifest = { + "schemaVersion": 1, + "status": "complete", + "sourceFile": str(self.source), + "cacheHash": self.hash, + "eventsPerSlice": 500000, + "devices": { + "0": { + "eventCount": 1, + "sliceCount": 1, + "readySlices": [0], + "slices": [ + { + "index": 0, + "startEventId": 0, + "endEventId": 0, + "file": "slice.db", + "ready": True, + } + ], + } + }, + } + seal_manifest(self.directory, self.manifest) + self.save() + + def save(self): + (self.directory / "manifest.json").write_text(json.dumps(self.manifest), encoding="utf-8") + + def test_readonly_transport_observes_writer_started_after_uri_creation(self): + from pt_snap_cli.core.dataset_files import member_is_immutable, readonly_member_uri + + uri = readonly_member_uri(self.db) + self.assertNotIn("immutable=1", uri) + self.assertFalse(member_is_immutable(self.db)) + with closing(sqlite3.connect(self.db)) as writer: + writer.execute("PRAGMA journal_mode=WAL") + writer.execute("UPDATE block_0 SET size=42") + writer.commit() + with closing(sqlite3.connect(uri, uri=True)) as reader: + self.assertEqual(reader.execute("SELECT size FROM block_0").fetchone(), (42,)) + + def test_consumer_write_after_sidecar_check_is_allowed(self): + from unittest.mock import patch + + from pt_snap_cli.core import snapshot_cache + + connect = sqlite3.connect + writers = [] + self.addCleanup(lambda: [writer.close() for writer in writers]) + + def open_during_write(*args, **kwargs): + # Start the consumer after preflight, when no sidecars existed. + writer = connect(self.db) + writers.append(writer) + writer.execute("PRAGMA journal_mode=WAL") + writer.execute("CREATE TABLE consumer_cache(value INTEGER)") + writer.commit() + return connect(*args, **kwargs) + + with patch.object(snapshot_cache.sqlite3, "connect", side_effect=open_during_write): + self.assertEqual(validate_cache(self.directory, self.hash), self.manifest) + + def test_base_write_after_sidecar_check_is_detected(self): + from unittest.mock import patch + + from pt_snap_cli.core import snapshot_cache + + connect = sqlite3.connect + writers = [] + self.addCleanup(lambda: [writer.close() for writer in writers]) + + def open_after_base_write(*args, **kwargs): + writer = connect(self.db) + writers.append(writer) + writer.execute("PRAGMA journal_mode=WAL") + writer.execute("UPDATE block_0 SET size=size+1") + writer.commit() + return connect(*args, **kwargs) + + with patch.object(snapshot_cache.sqlite3, "connect", side_effect=open_after_base_write): + with self.assertRaisesRegex(ValueError, "parser data"): + validate_cache(self.directory, self.hash) + + def test_consumer_commit_during_row_scan_is_allowed(self): + from unittest.mock import patch + + from pt_snap_cli.core import snapshot_cache + + with closing(sqlite3.connect(self.db)) as writer: + writer.execute("PRAGMA journal_mode=WAL") + encode = snapshot_cache._RowEncoder.encode + committed = False + + def encode_while_writing(encoder, row, inline_stack): + nonlocal committed + if not committed: + writer.execute("CREATE TABLE consumer_cache(value INTEGER)") + writer.commit() + writer.execute("PRAGMA wal_checkpoint(PASSIVE)") + committed = True + return encode(encoder, row, inline_stack) + + with patch.object(snapshot_cache._RowEncoder, "encode", encode_while_writing): + self.assertEqual(validate_cache(self.directory, self.hash), self.manifest) + self.assertTrue(committed) + + def test_parser_seal_does_not_override_producer_seal(self): + self.manifest["ptSnap"] = { + "memberHashKind": "parser-v1", + "members": {"slice.db": "0" * 64}, + } + self.save() + with self.assertRaisesRegex(ValueError, "member hash"): + validate_cache(self.directory, self.hash) + + def test_logical_producer_seal_without_parser_seal_is_checked(self): + self.manifest["ptSnap"] = { + "memberHashKind": "parser-v1", + "members": self.manifest.pop("parserCache")["members"], + } + self.save() + self.assertEqual(validate_cache(self.directory, self.hash), self.manifest) + with closing(sqlite3.connect(self.db)) as conn: + conn.execute("UPDATE block_0 SET size=size+1") + conn.commit() + with self.assertRaisesRegex(ValueError, "member hash"): + validate_cache(self.directory, self.hash) + + def test_legacy_producer_seal_is_not_overridden_by_parser_seal(self): + self.manifest["ptSnap"] = {"members": {"slice.db": source_hashes(self.db)[0]}} + self.save() + self.assertEqual(validate_cache(self.directory, self.hash), self.manifest) + with closing(sqlite3.connect(self.db)) as conn: + conn.execute("CREATE TABLE consumer_extra(value INTEGER)") + with self.assertRaisesRegex(ValueError, "member hash"): + validate_cache(self.directory, self.hash) + + def test_legacy_physical_seal_accepts_closed_wal_database(self): + with closing(sqlite3.connect(self.db)) as conn: + conn.execute("PRAGMA journal_mode=WAL") + self.manifest["ptSnap"] = {"members": {"slice.db": source_hashes(self.db)[0]}} + self.save() + self.assertEqual(validate_cache(self.directory, self.hash), self.manifest) + self.assertEqual(validate_cache(self.directory, self.hash), self.manifest) + + def test_legacy_physical_seal_rejects_committed_wal(self): + with closing(sqlite3.connect(self.db)) as conn: + conn.execute("PRAGMA journal_mode=WAL") + self.manifest["ptSnap"] = {"members": {"slice.db": source_hashes(self.db)[0]}} + self.save() + with closing(sqlite3.connect(self.db)) as writer: + writer.execute("CREATE TABLE consumer_cache(value INTEGER)") + writer.commit() + with self.assertRaisesRegex(ValueError, "sidecars"): + validate_cache(self.directory, self.hash) + + def test_source_hash_matches_shared_salt(self): + raw, salted = source_hashes(self.source) + self.assertEqual(raw, hashlib.sha256(self.source.read_bytes()).hexdigest()) + self.assertEqual( + salted, hashlib.sha256(b"mem_snapshot_parser_v2" + self.source.read_bytes()).hexdigest() + ) + + def test_extra_tables_indexes_and_dictionary_do_not_invalidate(self): + with closing(sqlite3.connect(self.db)) as conn, conn: + conn.executescript( + "CREATE TABLE memory_allocation_cache_v1_0(value INTEGER);" + "CREATE TABLE arbitrary_consumer_data(value TEXT);" + "CREATE INDEX consumer_index ON block_0(address);" + "ALTER TABLE trace_entry_0 ADD COLUMN consumer_note TEXT;" + "CREATE VIEW consumer_view AS SELECT id FROM block_0;" + "CREATE TRIGGER consumer_trigger AFTER UPDATE ON block_0 BEGIN SELECT 1; END;" + "INSERT INTO arbitrary_consumer_data VALUES('extra');" + "INSERT INTO dictionary VALUES('arbitrary_consumer_data','value','1','extra');" + ) + (self.directory / "parse.log").write_text("consumer log") + self.assertEqual(validate_cache(self.directory, self.hash), self.manifest) + with closing(sqlite3.connect(self.db)) as conn: + conn.execute("VACUUM") + self.assertEqual(validate_cache(self.directory, self.hash), self.manifest) + + def test_modified_base_data_is_rejected(self): + for table, assignment in ( + ("block_0", "size=size+1"), + ("trace_entry_0", "allocated=allocated+1"), + ("dictionary", "\"value\"='changed'"), + ): + with self.subTest(table=table): + with closing(sqlite3.connect(self.db)) as conn, conn: + conn.execute(f"UPDATE {table} SET {assignment}") + with self.assertRaises(ValueError): + validate_cache(self.directory, self.hash) + seal_manifest(self.directory, self.manifest) + self.save() + + def test_changed_source_is_rejected(self): + self.source.write_bytes(b"changed") + with self.assertRaises(ValueError): + validate_cache(self.directory, source_hashes(self.source)[1]) + + def test_closed_wal_cache_preserves_database_bytes(self): + with closing(sqlite3.connect(self.db)) as conn: + conn.execute("PRAGMA journal_mode=WAL") + before = self.db.read_bytes() + self.assertEqual(validate_cache(self.directory, self.hash), self.manifest) + self.assertEqual(self.db.read_bytes(), before) + self.assertEqual(validate_cache(self.directory, self.hash), self.manifest) + + def test_rollback_journal_is_rejected(self): + Path(str(self.db) + "-journal").write_bytes(b"") + with self.assertRaisesRegex(ValueError, "rollback journal"): + validate_cache(self.directory, self.hash) + + def test_active_wal_is_read_instead_of_ignoring_committed_rows(self): + with closing(sqlite3.connect(self.db)) as conn: + conn.execute("PRAGMA journal_mode=WAL") + conn.execute("UPDATE block_0 SET size=size+1") + conn.commit() + self.assertTrue(Path(str(self.db) + "-wal").exists()) + with self.assertRaises(ValueError): + validate_cache(self.directory, self.hash) + + def test_missing_or_corrupt_db_is_rejected_without_creating_it(self): + self.db.unlink() + with self.assertRaises(ValueError): + validate_cache(self.directory, self.hash) + self.assertFalse(self.db.exists()) + self.db.write_bytes(b"not a database") + with self.assertRaises(sqlite3.DatabaseError): + validate_cache(self.directory, self.hash) + + def test_incomplete_manifest_or_event_range_is_rejected(self): + for mutation in ( + lambda: self.manifest.update(status="building"), + lambda: self.manifest["devices"]["0"].update(readySlices=[]), + ): + mutation() + self.save() + with self.assertRaises(ValueError): + validate_cache(self.directory, self.hash) + self.manifest["status"] = "complete" + with closing(sqlite3.connect(self.db)) as conn, conn: + conn.execute("DELETE FROM trace_entry_0") + self.manifest["devices"]["0"]["readySlices"] = [0] + self.save() + with self.assertRaises(ValueError): + validate_cache(self.directory, self.hash) + + def test_missing_json_and_unsafe_member_are_rejected(self): + (self.directory / "manifest.json").unlink() + with self.assertRaises(OSError): + validate_cache(self.directory, self.hash) + self.manifest["devices"]["0"]["slices"][0]["file"] = "../outside.db" + self.save() + with self.assertRaises(ValueError): + validate_cache(self.directory, self.hash) + + def test_legacy_complete_artifact_is_structurally_validated(self): + del self.manifest["parserCache"] + self.save() + self.assertEqual(validate_cache(self.directory, self.hash), self.manifest) + + def test_parser_device_selection_does_not_hide_producer_selection(self): + self.manifest["ptSnap"] = {"identity": {"requestedDevice": 0}} + self.save() + with self.assertRaisesRegex(ValueError, "selected device"): + validate_cache(self.directory, self.hash) + self.assertEqual(validate_cache(self.directory, self.hash, device=0), self.manifest) + + def test_declared_device_must_match_inventory_and_request(self): + for selected in (1, True, "0", -1): + with self.subTest(selected=selected): + self.manifest["parserCache"]["requestedDevice"] = selected + self.save() + with self.assertRaises(ValueError): + validate_cache(self.directory, self.hash, device=0) + + def test_device_filtered_cache_cannot_satisfy_all_devices(self): + seal_manifest(self.directory, self.manifest, device=0) + self.save() + self.assertEqual(validate_cache(self.directory, self.hash, device=0), self.manifest) + for selection in (None, 1): + with self.assertRaises(ValueError): + validate_cache(self.directory, self.hash, device=selection) + + def test_invalid_manifest_shape_is_rejected(self): + for data in ( + [], + {**self.manifest, "parserCache": []}, + {**self.manifest, "schemaVersion": True}, + ): + (self.directory / "manifest.json").write_text(json.dumps(data)) + with self.assertRaises(ValueError): + validate_cache(self.directory, self.hash) + + def test_duplicate_json_keys_are_rejected(self): + (self.directory / "manifest.json").write_text('{"status":"building","status":"complete"}') + with self.assertRaises(ValueError): + validate_cache(self.directory, self.hash) + + def test_manifest_changed_during_validation_is_rejected(self): + from unittest.mock import patch + + from pt_snap_cli.core import snapshot_cache + + original = snapshot_cache.parser_members + + def change_manifest(directory, manifest, **kwargs): + members = original(directory, manifest, **kwargs) + self.manifest["consumerRevision"] = 1 + self.save() + return members + + with patch.object(snapshot_cache, "parser_members", side_effect=change_manifest): + with self.assertRaisesRegex(ValueError, "manifest changed"): + validate_cache(self.directory, self.hash) + + def test_parser_salt_changes_cache_identity(self): + from unittest.mock import patch + + from pt_snap_cli.core import snapshot_cache + + with patch.object(snapshot_cache, "PARSER_HASH_SALT", b"another-parser-version"): + with self.assertRaises(ValueError): + validate_cache(self.directory, source_hashes(self.source)[1]) + + def test_corrupt_secondary_index_is_rejected(self): + with closing(sqlite3.connect(self.db)) as conn: + conn.execute("CREATE INDEX block_address ON block_0(address)") + page_size = conn.execute("PRAGMA page_size").fetchone()[0] + root = conn.execute( + "SELECT rootpage FROM sqlite_master WHERE name='block_address'" + ).fetchone()[0] + with self.db.open("r+b") as file: + file.seek((root - 1) * page_size) + file.write(b"\xff" * page_size) + with self.assertRaises((ValueError, sqlite3.DatabaseError)): + validate_cache(self.directory, self.hash) + + def test_deep_manifest_is_a_validation_error(self): + (self.directory / "manifest.json").write_text("[" * 200000 + "0" + "]" * 200000) + with self.assertRaisesRegex(ValueError, "nesting"): + validate_cache(self.directory, self.hash) + + def test_legacy_dictionary_references_are_required(self): + del self.manifest["parserCache"] + for table in ("trace_entry_0", "block_0"): + with self.subTest(table=table): + with closing(sqlite3.connect(self.db)) as conn, conn: + original = conn.execute( + 'SELECT * FROM dictionary WHERE "table"=?', (table,) + ).fetchall() + conn.execute('DELETE FROM dictionary WHERE "table"=?', (table,)) + self.save() + with self.assertRaisesRegex(ValueError, "dictionary reference"): + validate_cache(self.directory, self.hash) + with closing(sqlite3.connect(self.db)) as conn, conn: + conn.executemany("INSERT INTO dictionary VALUES(?,?,?,?)", original) + + def test_cross_window_lifetime_references_remain_valid(self): + with closing(sqlite3.connect(self.db)) as conn, conn: + conn.execute("UPDATE block_0 SET allocEventId=-1, freeEventId=999") + seal_manifest(self.directory, self.manifest) + self.save() + self.assertEqual(validate_cache(self.directory, self.hash), self.manifest) diff --git a/tests/test_dataset_contract.py b/tests/test_dataset_contract.py index 2ceae85..34e7520 100644 --- a/tests/test_dataset_contract.py +++ b/tests/test_dataset_contract.py @@ -281,7 +281,6 @@ def test_multiple_devices_and_short_slices_are_legal(): ("ALTER TABLE block_0 RENAME TO block_1;", "device_tables"), ("CREATE TABLE callstack (id INTEGER, callstack TEXT);", "layout"), ("ALTER TABLE trace_entry_0 RENAME COLUMN callstack TO callstackId;", "columns"), - ("ALTER TABLE trace_entry_0 ADD COLUMN extra INTEGER;", "columns"), ( "ALTER TABLE trace_entry_0 RENAME TO saved; CREATE VIEW trace_entry_0 AS SELECT * FROM saved;", "table", @@ -314,6 +313,52 @@ def test_missing_slice_does_not_create_database(tmp_path): assert not path.exists() +def test_compatibility_appended_columns_do_not_relax_native_schema(tmp_path): + from pt_snap_cli.core.dataset_contract import TRACE_COLUMNS, _schema + + root = make_dataset(tmp_path / "dataset") + mutate_db(root, "ALTER TABLE trace_entry_0 ADD COLUMN consumer_note TEXT") + assert validate_dataset(root).real_event_count == 4 + with closing(sqlite3.connect(root / "device_0/slice_00001.db")) as conn: + with pytest.raises(DatasetContractError, match="columns"): + _schema(conn, "trace_entry_0", TRACE_COLUMNS, "native-member") + + +def test_unsealed_index_corruption_is_rejected_before_query(tmp_path): + from pt_snap_cli.core.dataset_resolver import DatasetResolver + from pt_snap_cli.core.errors import DatabaseSchemaError + + root = make_dataset(tmp_path / "dataset") + member = root / "device_0/slice_00001.db" + with closing(sqlite3.connect(member)) as conn: + conn.execute("CREATE INDEX consumer_index ON block_0(address)") + page_size = conn.execute("PRAGMA page_size").fetchone()[0] + page = conn.execute( + "SELECT rootpage FROM sqlite_master WHERE name='consumer_index'" + ).fetchone()[0] + with member.open("r+b") as source: + source.seek((page - 1) * page_size) + source.write(b"\xff" * page_size) + with pytest.raises(DatasetContractError, match="integrity|sqlite"): + validate_dataset(root) + with pytest.raises(DatabaseSchemaError, match="integrity|sqlite"): + DatasetResolver().inspect(root) + + +@pytest.mark.parametrize("resolver", [False, True]) +def test_deep_manifest_uses_domain_diagnostic(tmp_path, resolver): + from pt_snap_cli.core.dataset_resolver import DatasetResolver + from pt_snap_cli.core.errors import DatabaseSchemaError + + root = make_dataset(tmp_path / "dataset") + (root / "manifest.json").write_text("[" * 200000 + "0" + "]" * 200000) + with pytest.raises( + DatabaseSchemaError if resolver else DatasetContractError, + match="manifest nesting is too deep", + ): + DatasetResolver().inspect(root) if resolver else validate_dataset(root) + + def test_symlink_artifact_member_rejected(tmp_path): root = make_dataset(tmp_path / "dataset") path = root / "device_0/slice_00001.db" diff --git a/tests/test_dataset_focus.py b/tests/test_dataset_focus.py index 931843f..e8c2622 100644 --- a/tests/test_dataset_focus.py +++ b/tests/test_dataset_focus.py @@ -383,7 +383,8 @@ def forbidden(*args, **kwargs): monkeypatch.setattr(sqlite3, "connect", forbidden) with pytest.raises( - DatasetContractError if bare else DatabaseSchemaError, match="live SQLite sidecar" + DatasetContractError if bare else DatabaseSchemaError, + match="aliased SQLite sidecar" if dangling else "live SQLite sidecar", ): validate_dataset(root) if bare else DatasetResolver().inspect(root) assert before == hashes(root) @@ -395,7 +396,7 @@ def forbidden(*args, **kwargs): @pytest.mark.parametrize("manifest_path", [False, True]) @pytest.mark.parametrize("operation", ["inspect", "p0", "focus", "api", "cli"]) -def test_checkpointed_wal_read_without_artifact_changes( +def test_checkpointed_wal_read_preserves_database_bytes( tmp_path, monkeypatch, manifest_path, operation ): root = make_dataset(tmp_path / "dataset") @@ -408,7 +409,6 @@ def test_checkpointed_wal_read_without_artifact_changes( assert source.read(20)[18:20] == b"\x02\x02" assert not list(root.rglob("*-wal")) and not list(root.rglob("*-shm")) before = hashes(root) - inventory = sorted(str(p.relative_to(root)) for p in root.rglob("*")) opened = [] original = sqlite3.connect @@ -445,10 +445,15 @@ def connect(*args, **kwargs): ) assert result.exit_code == 0, result.output assert json.loads(result.output)["rows"][0]["id"] == 5 - assert before == hashes(root) - assert inventory == sorted(str(p.relative_to(root)) for p in root.rglob("*")) - assert opened and all("mode=ro&immutable=1" in str(uri) for uri in opened) - assert not list(root.rglob("*-wal")) and not list(root.rglob("*-shm")) + after = hashes(root) + assert before == {name: after[name] for name in before} + # SQLite's read-only WAL protocol may create transport sidecars, but must + # neither modify a source DB/manifest nor commit any pages to the WAL. + relative = str(member.relative_to(root)) + assert set(after) - set(before) <= {relative + "-wal", relative + "-shm"} + wal = Path(str(member) + "-wal") + assert not wal.exists() or wal.stat().st_size == 0 + assert opened and all("mode=ro" in str(uri) and "immutable=1" not in str(uri) for uri in opened) @pytest.mark.parametrize("alias_parent", [False, True]) diff --git a/tests/test_interop_acceptance.py b/tests/test_interop_acceptance.py index 6c7d796..ec050a7 100644 --- a/tests/test_interop_acceptance.py +++ b/tests/test_interop_acceptance.py @@ -163,9 +163,16 @@ def test_compatible_owned_cache_reuse_and_external_force_preserve_actual_bytes(t raw.pop("ptSnap") # external original-shaped cache, not our ownership attestation manifest.write_text(json.dumps(raw)) external = inventory(result.db_path) + reused = service.import_snapshot(dataclasses.replace(request, force=True)) + assert reused.reused and reused.metadata is None + assert inventory(result.db_path) == external + # Invalid external artifacts must still be preserved, including under force. + with closing(sqlite3.connect(result.db_path / "device_0/slice_00000.db")) as conn, conn: + conn.execute("UPDATE block_0 SET size=size+1") + invalid = inventory(result.db_path) with pytest.raises(ImportExecutionError, match="preserv"): service.import_snapshot(dataclasses.replace(request, force=True)) - assert inventory(result.db_path) == external + assert inventory(result.db_path) == invalid assert before != external