diff --git a/README.md b/README.md index 2f28e83..f4a8471 100644 --- a/README.md +++ b/README.md @@ -52,6 +52,39 @@ deterministic names, replay validation, and failure-safe publication. See the [full quick start guide](docs/en/quickstart.md) for a walkthrough. +Complete compatibility-v1 sharded artifacts can be focused by directory or +`manifest.json`, without the archived pickle. See [dataset focus and addressing](docs/en/sharded-snapshotdb.md#complete-dataset-focus-and-addressing-p1) +for read-only overview, event routing, `--slice`, and explicit cross-slice limits. + +Opt into a **native, not msinsight-compatible** dataset with +`pt-snap import snapshot.pkl --events-per-slice 50000 --output-dir captures --json`. +The output is `captures/snapshot.pkl.pt-snap-native-v2/`; default standalone import +is unchanged. Reuse validates all shards and content/options/semantic versions. +Nonmatching recognized targets require `--force`; unknown targets are preserved. +Publication/focus failures are compensated, with explicit recovery evidence if +rollback fails (force is not a single crash-atomic swap). See the +[native import guide](docs/en/quickstart.md#optional-import-a-native-sharded-dataset). +Dataset `event` addressing and point-event `active_blocks_at_event` / +`active_memory_callstack_at_event` share batched cross-shard alloc/free sources, +including original artifacts without extensions. Built-in global peaks, event pages, +canonical stack statistics, deduplicated lifecycles, terminal leak candidates and +`report peak-memory` share bounded core semantics; custom SQL/overrides remain unsupported +on datasets. See the [global support matrix](docs/en/querying.md#dataset-global-built-in-support-p3) +and [dataset attribution and coverage](docs/en/querying.md#dataset-point-event-attribution). + +Explicit `pt-snap import snapshot.pkl --format msinsight --json` produces +`snapshot.pkl.msinsight/` for the fixed msinsight revision's **same original +pickle + adjacent cache** entrance. Inline stacks cost more space than native +interning. Existing compatible targets are never replaced, even with `--force`; +only fully validated identical pt-snap caches are reused. GUI acceptance remains +**pending/not run**; hash equality alone is not acceptance. See +[compatible export and limits](docs/en/sharded-snapshotdb.md#explicit-msinsight-compatible-export-p2). + +See [offline interoperability acceptance and bounded performance](docs/en/interop-acceptance.md) +for the two fixed-version chains, deterministic reference tools, honest three-mode +measurements and the explicitly **pending/not-run GUI checklist**. Diagnostic skills +accept explicitly selected complete validated datasets, not arbitrary directories. + ## Commands | Command | Description | @@ -98,6 +131,7 @@ See the [documentation index](docs/README.md) for all English and Chinese guides | Splitting snapshots | [Splitting Snapshots](docs/en/splitting.md) | | Agent skills | [Agent Skills](docs/en/skills.md) | | Database format | [SnapshotDB Schema](docs/en/database.md) | +| Sharded protocols, native import and compatibility limits | [Sharded SnapshotDB protocol](docs/en/sharded-snapshotdb.md) | | Python API | [SnapshotAnalyzer API](docs/en/snapshot-analyzer-api.md) | | Result mapping utility | [ResultMapper API](docs/en/result-mapper-api.md) | diff --git a/README_zh.md b/README_zh.md index 264544d..b7c1ddc 100644 --- a/README_zh.md +++ b/README_zh.md @@ -47,6 +47,32 @@ pt-snap split snapshot.pkl --slices 4 --output snapshot-slices 完整的入门指南见 [Quick Start](docs/zh/quickstart.md)。 +完整兼容 v1 分片产物可以目录或 `manifest.json` 为 focus,无需已归档 pickle。 +只读 overview、事件路由、`--slice` 和跨片限制见 +[完整数据集 focus 与寻址](docs/zh/sharded-snapshotdb.md#完整数据集-focus-与寻址p1)。 + +用 `pt-snap import snapshot.pkl --events-per-slice 50000 --output-dir captures --json` +显式生成**原生、非 msinsight 兼容**数据集 `captures/snapshot.pkl.pt-snap-native-v2/`。 +默认单库导入不变;复用校验全部片及内容/选项/语义版本。不匹配的已识别目标须 +`--force`,未知目标保留。发布/focus 失败会补偿,回滚自身失败则保留并报告恢复证据 +(force 不是单次崩溃原子 swap)。详见 +[原生分库导入](docs/zh/quickstart.md#可选导入原生分库数据集)。数据集 `event` 寻址及定点 +`active_blocks_at_event` / `active_memory_callstack_at_event` 共用跨片 alloc/free 批量来源, +原版无扩展产物也适用。内建全局峰值、事件分页、canonical 栈统计、去重生命周期、 +终片泄漏候选及 `report peak-memory` 共用有界 core 语义;数据集仍不支持自定义 SQL/override。 +详见[全局支持矩阵](docs/zh/querying.md#数据集全局内建支持p3)及 +[数据集定点归因与覆盖](docs/zh/querying.md#数据集定点事件归因)。 + +显式 `pt-snap import snapshot.pkl --format msinsight --json` 生成 +`snapshot.pkl.msinsight/`,对齐固定 msinsight 版本的**同一原始 pickle + 邻接缓存**入口。 +内联栈比原生去重占更多空间;即使 `--force` 也不替换已有兼容目标,只复用完整校验且 +身份相同的 pt-snap 缓存。GUI **待验收/未运行**,hash 相等不代表验收通过。详见 +[兼容导出及限制](docs/zh/sharded-snapshotdb.md#显式-msinsight-兼容导出p2)。 + +[离线双向验收与有界性能](docs/zh/interop-acceptance.md) 提供固定版本两条链路、 +确定性独立参考工具、真实三模式测量及明确**待验收/not-run 的 GUI 清单**。 +诊断 skills 接受显式选择的完整已校验数据集,不接受任意目录。 + ## 命令 | 命令 | 说明 | @@ -91,6 +117,7 @@ helper 会先路由采集,无需先安装 CLI。分析仍需要 CLI 和 Snapsh | 拆分快照 | [拆分快照](docs/zh/splitting.md) | | Agent skill | [Agent Skills](docs/zh/skills.md) | | 数据库格式 | [SnapshotDB Schema](docs/zh/database.md) | +| 分片协议、原生导入与兼容边界 | [分片 SnapshotDB 协议](docs/zh/sharded-snapshotdb.md) | | Python API | [SnapshotAnalyzer API](docs/zh/snapshot-analyzer-api.md) | | 结果映射工具 | [ResultMapper API](docs/zh/result-mapper-api.md) | diff --git a/benchmarks/dataset_import_baseline.py b/benchmarks/dataset_import_baseline.py new file mode 100644 index 0000000..ce61b4f --- /dev/null +++ b/benchmarks/dataset_import_baseline.py @@ -0,0 +1,395 @@ +"""Opt-in SAME-source three-mode baseline in isolated children and NEW outputs. + +No warmup, downloads, installs, original producer, GUI or cleanup. Trusted pickle +loads require a separate provenance gate and explicit --trusted-pickle consent. +""" + +from __future__ import annotations + +import argparse +import functools +import hashlib +import json +import logging +import os +import platform +import resource +import sqlite3 +import subprocess +import sys +import time +import uuid +from collections import defaultdict +from contextlib import closing +from pathlib import Path + +MODES = ("single-db", "pt-snap-native-v2", "compatibility-v1") + + +def rss_kib(raw: float, system: str) -> float: + return raw / 1024 if system == "darwin" else raw + + +def source_identity(source: Path, expected: str) -> dict: + if source.is_symlink() or source.resolve() != source or not source.is_file(): + raise ValueError("Source must be an explicit canonical regular file") + if ( + not isinstance(expected, str) + or len(expected) != 64 + or any(c not in "0123456789abcdef" for c in expected) + ): + raise ValueError("Require a lowercase SHA256 identity") + if source.stat().st_size > 2 * 1024 * 1024: + raise ValueError("Require the exact reviewed small source (<=2 MiB)") + digest, size = hashlib.sha256(), 0 + with source.open("rb") as file: + for chunk in iter(lambda: file.read(1024 * 1024), b""): + size += len(chunk) + if size > 2 * 1024 * 1024: + raise ValueError("Source grew beyond the small input bound") + digest.update(chunk) + if digest.hexdigest() != expected: + raise ValueError("Require the exact reviewed small source (<=2 MiB)") + return {"path": str(source), "sha256": expected, "bytes": size} + + +def instrument(module, name: str, label: str, phases: dict) -> None: + original = getattr(module, name) + + @functools.wraps(original) + def measured(*args, **kwargs): + start = time.perf_counter() + try: + return original(*args, **kwargs) + finally: + phases[label]["seconds"] += time.perf_counter() - start + phases[label]["calls"] += 1 + + setattr(module, name, measured) # child-local instrumentation only + + +def cache_hashes(output: Path) -> dict: + result = {} + for index, path in enumerate(output.rglob("*")): + if index >= 256 or path.is_symlink() or path.resolve() != path: + raise ValueError("Unknown/unsafe benchmark output member") + if path.is_dir(): + continue + if not path.is_file() or path.name.endswith(("-wal", "-shm", "-journal")): + raise ValueError("Benchmark output must be finalized sidecar-free files") + digest = hashlib.sha256() + with path.open("rb") as file: + for chunk in iter(lambda: file.read(1024 * 1024), b""): + digest.update(chunk) + result[path.relative_to(output).as_posix()] = digest.hexdigest() + return result + + +def require_owned_child(args, identity: dict) -> None: + envelope = args.envelope + if ( + envelope is None + or envelope != args.output.parent / "run-envelope.json" + or envelope.is_symlink() + or envelope.resolve() != envelope + or not envelope.is_file() + or envelope.stat().st_size > 1024 * 1024 + ): + raise ValueError("Hidden child requires the bounded same-run ownership envelope") + with envelope.open("rb") as file: + raw = file.read(1024 * 1024 + 1) + if len(raw) > 1024 * 1024: + raise ValueError("Oversized ownership envelope") + owner = json.loads(raw) + if ( + not isinstance(owner, dict) + or owner.get("source") != identity + or owner.get("device") != args.device + or owner.get("capacity") != args.capacity + or args.output.name != args.mode + ): + raise ValueError("Child source/options/output do not match this run") + recorded = owner["cacheHashes"].get(args.mode) + if args.phase == "cold": + if recorded is not None or any(args.output.iterdir()): + raise ValueError("Cold child requires an empty exclusively allocated output") + elif recorded is None or cache_hashes(args.output) != recorded: + raise ValueError("Reuse child requires unchanged successful owned cold output") + + +def child(args) -> dict: + from pt_snap_cli.api import SnapshotAnalyzer + from pt_snap_cli.core import import_service, sharded_replay_service + from pt_snap_cli.core.import_service import ImportService + from pt_snap_cli.core.models import ImportOptions + from pt_snap_cli.snapshot.tools.adaptors import snapshot2db + + logging.disable(logging.CRITICAL) + identity = source_identity(args.source, args.sha256) + require_owned_child(args, identity) # BEFORE any import/pickle execution + phases = defaultdict(lambda: {"seconds": 0.0, "calls": 0}) + instrument(snapshot2db, "load_snapshot_representation", "load_representation", phases) + instrument( + sharded_replay_service, "load_snapshot_representation", "load_representation", phases + ) + instrument(snapshot2db, "replay_snapshot", "standalone_replay_with_sqlite_hooks", phases) + instrument( + sharded_replay_service, "build_shards", "sharded_replay_with_sqlite_and_backfill", phases + ) + instrument(import_service, "convert_shards", "compatible_conversion", phases) + options = ImportOptions( + args.source, + args.output, + device=args.device, + set_focus=False, + force=False, + events_per_slice=None if args.mode == "single-db" else args.capacity, + format=args.mode, + ) + start = time.perf_counter() + service = ImportService() + if args.phase == "reuse": + + def no_rebuild(*_args, **_kwargs): + raise AssertionError("Reuse cache miss: refuse deserialization/rebuild/overwrite") + + service._backend.dump_to_db = no_rebuild + service._dataset_backend.dump_to_dataset = no_rebuild + service._msinsight_backend.dump_to_dataset = no_rebuild + result = service.import_snapshot(options) + wall = time.perf_counter() - start + usage = resource.getrusage(resource.RUSAGE_SELF) # this process ONLY, before analysis + if result.reused != (args.phase == "reuse"): + raise AssertionError("Cold/reuse measurement did not follow requested cache path") + members = [result.db_path] if result.db_path.is_file() else sorted(result.db_path.rglob("*.db")) + disk = { + str(p.relative_to(args.output)): p.stat().st_size + for p in args.output.rglob("*") + if p.is_file() + } + raw_blocks = 0 + unique = set() + for member in members: + with closing(sqlite3.connect(member.as_uri() + "?mode=ro&immutable=1", uri=True)) as conn: + for row in conn.execute( + f"SELECT id,address,size,requestedSize,allocEventId,freeEventId FROM block_{args.device}" + ): + raw_blocks += 1 + unique.add(tuple(row)) + query_rows = {} + with SnapshotAnalyzer(result.db_path) as analyzer: + capabilities = analyzer.list_capabilities()["templates"] + ids = analyzer.get_database_overview()["devices"] + last = next(d["last_event_id"] for d in ids if d["device_id"] == args.device) + params = { + "event": {"id": last}, + "active_blocks_at_event": {"event_id": last}, + "active_memory_callstack_at_event": {"event_id": last, "top_n": -1}, + "preexisting_live": {"event_id": last}, + "callstack_analysis": {"min_count": 1}, + } + for template in capabilities: + name = template["name"] + if name not in ( + "event", + "allocation", + "memory_peak", + "allocator_gap", + "active_blocks_at_event", + "active_memory_callstack_at_event", + "preexisting_live", + "block", + "freed_block_lifetime", + "leak_detection", + "callstack_analysis", + ): + continue + start = time.perf_counter() + query = analyzer.execute_query( + name, params.get(name, {}), device_id=args.device, exact_total=True, timeout_s=30 + ) + query_rows[name] = { + "seconds": time.perf_counter() - start, + "returned": query["returned"], + "total": query["total"], + "has_more": query["has_more"], + "truncated": query["truncated"], + "scope": query.get("scope"), + "dataset_support": template["dataset_support"], + "status": "passed", + } + if source_identity(args.source, args.sha256) != identity: + raise AssertionError("Source changed during measurement") + return { + "status": "passed", + "mode": args.mode, + "phase": args.phase, + "import_wall_seconds": wall, + "isolated_import_process_peak_rss_raw": usage.ru_maxrss, + "isolated_import_process_peak_rss_kib": rss_kib(usage.ru_maxrss, sys.platform), + "isolated_import_process_peak_rss_mib": rss_kib(usage.ru_maxrss, sys.platform) / 1024, + "rss_scope": "RUSAGE_SELF fresh child high-water through import, before queries; includes interpreter/imports", + "phase_timings": dict(phases), + "phase_timing_scope": "named wrapped functions, inclusive; sharded replay includes SQLite writes/backfills, NOT parser-only", + "phase_rss": { + "status": "unavailable", + "reason": "No per-phase sampler/isolation; cumulative high-waters cannot be subtracted into load/replay peaks", + }, + "other_phase_timings": { + "status": "unavailable", + "reason": "validation/hash/metadata/publication remainder not separately instrumented", + }, + "reused": result.reused, + "source": identity, + "device": args.device, + "capacity": args.capacity if args.mode != "single-db" else None, + "artifact": str(result.db_path), + "total_disk_bytes": sum(disk.values()), + "individual_files_bytes": disk, + "individual_shard_bytes": { + str(p.relative_to(args.output)): p.stat().st_size for p in members + }, + "raw_block_rows": raw_blocks, + "unique_block_invariants": len(unique), + "repeated_block_rows": raw_blocks - len(unique), + "repeated_block_cost": { + "repeated_base_integer_cells": (raw_blocks - len(unique)) * 7, + "physical_disk_attribution": "unavailable: shared SQLite pages/indexes cannot be apportioned to duplicated rows by file-size subtraction", + }, + "queries": query_rows, + "errors": [], + "python": sys.executable, + "python_version": sys.version, + "platform": platform.platform(), + "source_module": import_service.__file__, + } + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--source", required=True, type=Path) + parser.add_argument("--sha256", required=True) + parser.add_argument("--device", required=True, type=int) + parser.add_argument("--capacity", required=True, type=int) + parser.add_argument("--output", required=True, type=Path) + parser.add_argument("--trusted-pickle", required=True, action="store_true") + parser.add_argument("--mode", choices=MODES, help=argparse.SUPPRESS) + parser.add_argument("--phase", choices=("cold", "reuse"), help=argparse.SUPPRESS) + parser.add_argument("--envelope", type=Path, help=argparse.SUPPRESS) + args = parser.parse_args() + if (args.mode is None) != (args.phase is None) or (args.mode is None) != ( + args.envelope is None + ): + parser.error("Hidden mode/phase/envelope must be supplied together") + args.source = args.source.absolute() + args.output = args.output.absolute() + identity = source_identity(args.source, args.sha256) + if ( + args.device < 0 + or args.capacity <= 0 + or args.output.is_symlink() + or args.output.resolve() != args.output + ): + raise ValueError("Canonical output, nonnegative device and positive capacity required") + if args.mode: + print(json.dumps(child(args))) + return 0 + args.output.mkdir() # exclusive parent; no adoption, replacement or cleanup + envelope = args.output / "run-envelope.json" + owner = { + "runId": uuid.uuid4().hex, + "source": identity, + "device": args.device, + "capacity": args.capacity, + "cacheHashes": {}, + } + with envelope.open("x") as file: + json.dump(owner, file) + records = [] + env = {**os.environ, "PYTHONDONTWRITEBYTECODE": "1"} + script = Path(__file__).resolve() + head = subprocess.run( + ["git", "rev-parse", "HEAD"], check=True, capture_output=True, text=True + ).stdout.strip() + source_diff = subprocess.run( + ["git", "diff", "--binary", "HEAD"], check=True, capture_output=True + ).stdout + status = "passed" + for mode in MODES: + output = args.output / mode + output.mkdir() + for phase in ("cold", "reuse"): + command = [ + sys.executable, + str(script), + "--source", + str(args.source), + "--sha256", + args.sha256, + "--device", + str(args.device), + "--capacity", + str(args.capacity), + "--output", + str(output), + "--trusted-pickle", + "--mode", + mode, + "--phase", + phase, + "--envelope", + str(envelope), + ] + start = time.perf_counter() + result = subprocess.run(command, capture_output=True, text=True, env=env) + (args.output / f"{mode}-{phase}.stdout.log").write_text(result.stdout) + (args.output / f"{mode}-{phase}.stderr.log").write_text(result.stderr) + record = { + "command": command, + "cwd": str(Path.cwd()), + "env": {k: env[k] for k in ("PYTHONPATH", "PYTHONDONTWRITEBYTECODE") if k in env}, + "exit": result.returncode, + "status": "passed" if result.returncode == 0 else "failed", + "subprocess_wall_seconds": time.perf_counter() - start, + } + if result.returncode == 0: + record["results"] = json.loads(result.stdout) + if phase == "cold": + owner["cacheHashes"][mode] = cache_hashes(output) + with envelope.open("w") as file: + json.dump(owner, file) + elif cache_hashes(output) != owner["cacheHashes"][mode]: + raise AssertionError("Reuse changed owned artifact bytes") + else: + record["error"] = result.stderr + status = "failed" + records.append(record) + if status == "failed": + break + if status == "failed": + break + report = { + "status": status, + "source_head": head, + "tracked_diff_sha256": hashlib.sha256(source_diff).hexdigest(), + "benchmark_script_sha256": hashlib.sha256(script.read_bytes()).hexdigest(), + "records": records, + "cold_definition": "fresh output/process, no warmup; OS page cache NOT cleared", + "reuse_definition": "fresh process reusing only this run's recognized owned artifact, force=False", + "original_producer": "not-run; preexisting single cold wall is not comparable RSS/phase evidence", + "GUI": "pending/not-run", + "limits": [ + "Pickle loads in full memory; sharding is not streaming", + "Shards repeat long-lived blocks; compatible format repeats inline stack text", + "No unstable elapsed-time assertion or performance benefit promise", + "Query latency includes validation/member hashing and repeated scans; only one local observation", + ], + } + with (args.output / "performance.json").open("x") as file: + json.dump(report, file, indent=2) + print(json.dumps({"status": status, "records": len(records), "output": str(args.output)})) + return 0 if status == "passed" else 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/benchmarks/interop_acceptance.py b/benchmarks/interop_acceptance.py new file mode 100644 index 0000000..0f1182c --- /dev/null +++ b/benchmarks/interop_acceptance.py @@ -0,0 +1,774 @@ +"""Bounded offline physical oracle; never downloads, loads pickle or runs a producer/GUI. + +The explicit receipt is caller-reviewed evidence, not a proof of producer authenticity. +The small reference materializes rows in memory: it is NOT a product RSS contract. +""" + +from __future__ import annotations + +import argparse +import hashlib +import json +import os +import sqlite3 +import subprocess +import sys +from collections import Counter, defaultdict +from contextlib import closing +from pathlib import Path + +REVISION = "101f65b877a267ffd5f66ea3834706057ba243e5" +TRACE = ( + "id", + "action", + "address", + "size", + "stream", + "allocated", + "active", + "reserved", + "callstack", +) +BLOCK = ("id", "address", "size", "requestedSize", "state", "allocEventId", "freeEventId") +INVARIANT = tuple(k for k in BLOCK if k != "state") +METRICS = ("allocated", "active", "reserved") + + +def canonical(path: Path) -> Path: + path = path.absolute() + if ".." in path.parts: + raise ValueError("Traversal is not a canonical path") + for current in (path, *path.parents): + if current.is_symlink() or (current.exists() and current.resolve() != current): + raise ValueError(f"Noncanonical/symlink input: {current}") + return path + + +def hash_file(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as file: + for chunk in iter(lambda: file.read(1024 * 1024), b""): + digest.update(chunk) + return digest.hexdigest() + + +def read_json(path: Path) -> dict: + path = canonical(path) + if not path.is_file() or path.stat().st_size > 1024 * 1024: + raise ValueError("Require a regular JSON file of at most 1 MiB") + + def unique(pairs): + result = {} + for key, value in pairs: + if key in result: + raise ValueError("Duplicate JSON key") + result[key] = value + return result + + def invalid(value): + raise ValueError(f"Nonfinite JSON value: {value}") + + with path.open("rb") as file: + raw = file.read(1024 * 1024 + 1) + if len(raw) > 1024 * 1024: + raise ValueError("JSON grew beyond the 1 MiB bound") + result = json.loads(raw, object_pairs_hook=unique, parse_constant=invalid) + if not isinstance(result, dict): + raise ValueError("Require a JSON object") + return result + + +def disjoint_output(root: Path, output: Path) -> None: + if output == root or root in output.parents or output in root.parents: + raise ValueError("Acceptance output must not overlap an input artifact") + + +def inventory(root: Path) -> dict[str, str]: + """Inspect all members, including unexpected/dangling sidecars, before SQLite.""" + root = canonical(root) + if not root.is_dir(): + raise ValueError("Artifact must be an existing directory") + result = {} + total = 0 + for index, path in enumerate(root.rglob("*")): + if index >= 256: + raise ValueError("Reference exceeds the 256-entry inventory bound") + canonical(path) + if path.is_dir(): + continue + if not path.is_file() or path.name.endswith(("-wal", "-shm", "-journal")): + raise ValueError(f"Unexpected member/sidecar: {path}") + total += path.stat().st_size + if total > 32 * 1024 * 1024: + raise ValueError("Reference exceeds this small oracle's 32 MiB input bound") + result[path.relative_to(root).as_posix()] = hash_file(path) + return result + + +def verify_receipt(root: Path, receipt: dict, source: Path | None = None) -> dict[str, str]: + if receipt.get("revision") != REVISION: + raise ValueError("Require the exact reviewed upstream revision") + actual = inventory(root) + if actual != receipt.get("artifactHashes"): + raise ValueError("Artifact member set or checksum differs from reviewed receipt") + if source is not None: + source = canonical(source) + if not source.is_file() or source.stat().st_size > 2 * 1024 * 1024: + raise ValueError("Source exceeds the reviewed small hash-only input bound") + if hash_file(source) != receipt.get("fixtureSHA256"): + raise ValueError("Source checksum differs from reviewed receipt") + if source.stat().st_size != receipt.get("fixtureBytes"): + raise ValueError("Source size differs from reviewed receipt") + return actual + + +def physical_oracle(root: Path, device: int) -> dict: + """Independent immutable SQL + primitive comparisons; no pt-snap imports here.""" + before = inventory(root) + manifest = read_json(root / "manifest.json") + if manifest.get("schemaVersion") != 1 or manifest.get("format", "compatibility-v1") not in ( + "compatibility-v1", + "pt-snap-native-v2", + ): + raise ValueError("Unknown artifact version/format") + if manifest["status"] != "complete" or type(device) is not int or device < 0: + raise ValueError("Require a complete artifact and explicit nonnegative device") + slices = manifest["devices"][str(device)]["slices"] + if not isinstance(slices, list) or not 0 < len(slices) <= 64: + raise ValueError("Require between 1 and 64 physical slices") + events, lifecycles, observations, dictionaries, schemas = {}, {}, [], [], [] + boundaries = 0 + trace_rows = 0 + raw_blocks = 0 + for item in slices: + path = canonical(root / item["file"]) + if path.relative_to(root).as_posix() not in before or not item["ready"]: + raise ValueError("Unreviewed/unready shard") + with closing(sqlite3.connect(path.as_uri() + "?mode=ro&immutable=1", uri=True)) as conn: + conn.row_factory = sqlite3.Row + columns = [r[1] for r in conn.execute(f"PRAGMA table_info(trace_entry_{device})")] + sql = ( + f"SELECT t.*,c.callstack FROM trace_entry_{device} t LEFT JOIN callstack c ON t.callstackId=c.id" + if "callstackId" in columns + else f"SELECT * FROM trace_entry_{device}" + ) + real = [] + for record in conn.execute(sql + " LIMIT 20001"): + trace_rows += 1 + if trace_rows > 20_000: + raise ValueError( + "Reference exceeds 20000 physical trace rows, including boundaries" + ) + row = {k: record[k] for k in TRACE} + if row["id"] < 0: + boundaries += 1 + else: + if row["id"] in events: + raise ValueError("Duplicate real ID across physical shards") + events[row["id"]] = row + real.append(row["id"]) + if len(events) > 20_000: + raise ValueError("Reference exceeds the 20000 real-event oracle bound") + if not real or min(real) != item["startEventId"] or max(real) != item["endEventId"]: + raise ValueError("Physical real range does not match manifest") + blocks = [] + for record in conn.execute(f"SELECT * FROM block_{device} LIMIT 20001"): + raw_blocks += 1 + if raw_blocks > 20_000: + raise ValueError("Reference exceeds the 20000 physical-block oracle bound") + blocks.append(dict(record)) + observations.append({r["id"]: r for r in blocks}) + for row in blocks: + old = lifecycles.get(row["id"]) + if old is not None and any(old[k] != row[k] for k in INVARIANT): + raise ValueError( + "Ambiguous physical lifetime identity; do not deduplicate by address" + ) + lifecycles[row["id"]] = row # latest shard observation, not state at E + dictionary = [] + for record in conn.execute("SELECT * FROM dictionary LIMIT 257"): + if len(dictionary) >= 256: + raise ValueError("Reference dictionary exceeds 256 rows per shard") + dictionary.append(tuple(record)) + dictionaries.append(sorted(dictionary)) + schema = {} + for name in (f"trace_entry_{device}", f"block_{device}", "dictionary"): + columns = [] + for record in conn.execute(f"PRAGMA table_info({name})"): + if len(columns) >= 64: + raise ValueError("Reference schema exceeds 64 columns per table") + columns.append(tuple(record)) + schema[name] = columns + schemas.append(schema) + if len(events) != manifest["devices"][str(device)]["eventCount"]: + raise ValueError("Physical event count does not match manifest") + if inventory(root) != before: + raise ValueError("Artifact changed during physical reference read") + return { + "events": events, + "blocks": lifecycles, + "observations": observations, + "slices": slices, + "boundaries": boundaries, + "raw_blocks": raw_blocks, + "dictionaries": dictionaries, + "schemas": schemas, + } + + +def canonical_database(oracle: dict, target: Path, device: int) -> None: + """Create only a NEW standalone reference from physical rows, not a pickle replay.""" + target = canonical(target) + with target.open("xb"): + pass + with closing(sqlite3.connect(target)) as conn, conn: + conn.executescript( + 'CREATE TABLE dictionary ("table" TEXT,"column" TEXT,"key" TEXT,"value" TEXT);' + f"CREATE TABLE trace_entry_{device} (id INTEGER PRIMARY KEY,action INTEGER,address INTEGER,size INTEGER,stream INTEGER,allocated INTEGER,active INTEGER,reserved INTEGER,callstack TEXT);" + f"CREATE TABLE block_{device} (id INTEGER PRIMARY KEY,address INTEGER,size INTEGER,requestedSize INTEGER,state INTEGER,allocEventId INTEGER,freeEventId INTEGER);" + ) + conn.executemany( + f"INSERT INTO trace_entry_{device} VALUES (?,?,?,?,?,?,?,?,?)", + [tuple(r[k] for k in TRACE) for r in oracle["events"].values()], + ) + conn.executemany( + f"INSERT INTO block_{device} VALUES (?,?,?,?,?,?,?)", + [tuple(r[k] for k in BLOCK) for r in oracle["blocks"].values()], + ) + conn.executemany("INSERT INTO dictionary VALUES (?,?,?,?)", oracle["dictionaries"][0]) + + +def require(condition: bool, name: str) -> None: + if not condition: + raise AssertionError(f"Differential failed: {name}") + + +def compare_artifact(root: Path, device: int, reference: Path) -> dict: + from pt_snap_cli.api import SnapshotAnalyzer + from pt_snap_cli.core.report_service import ReportService + + root = canonical(root) + reference = canonical(reference) + disjoint_output(root, reference) + before = inventory(root) + oracle = physical_oracle(root, device) + canonical_database(oracle, reference, device) + events, blocks = oracle["events"], oracle["blocks"] + records = [] + points = sorted( + { + min(events), + max(events), + *(i[k] for i in oracle["slices"] for k in ("startEventId", "endEventId")), + *(min(events, key=lambda n: (-events[n][m], n)) for m in METRICS), + } + ) + with SnapshotAnalyzer(root) as actual, SnapshotAnalyzer(reference) as single: + + def query(name, params=None, **kwargs): + result = actual.execute_query( + name, params, device_id=device, exact_total=True, timeout_s=30, **kwargs + ) + scope = result["scope"] + coverage = scope if scope["kind"] != "event" else scope["source_coverage"] + require( + coverage["range_complete"] and not scope["boundary_events_included"], + name + " scope", + ) + declared = {r["column"] for r in actual.get_template_info(name)["output_schema"]} + require( + all(set(row) <= declared for row in result["rows"]), name + " declared row keys" + ) + return result + + trace = query("event") + require( + trace["rows"] == sorted(events.values(), key=lambda r: r["id"]), + "ALL nine-column real events/actions/streams/counters/text", + ) + require(trace["total"] == len(events) and not trace["truncated"], "event exact total") + allocation = query("allocation") + require( + allocation["rows"] + == [{k: events[n][k] for k in ("id", *METRICS)} for n in sorted(events)], + "allocation counters", + ) + records.extend(("event", "allocation")) + ranges = [ + (min(events), max(events)), + *((i["startEventId"], i["endEventId"]) for i in oracle["slices"]), + ] + for lo, hi in ranges: + params = {"start_id": lo, "end_id": hi} + peak = query("memory_peak", params)["rows"][0] + gap = query("allocator_gap", params)["rows"][0] + for metric in METRICS: + event = min( + (r for r in events.values() if lo <= r["id"] <= hi), + key=lambda r: (-r[metric], r["id"]), + ) + require( + (peak["peak_" + metric], peak["peak_" + metric + "_event_id"]) + == (event[metric], event["id"]), + "global peak earliest ties", + ) + require( + gap["reserved_active_gap_at_" + metric + "_peak"] + == event["reserved"] - event["active"], + "same-event active gap", + ) + require( + gap["reserved_allocated_gap_at_" + metric + "_peak"] + == event["reserved"] - event["allocated"], + "same-event allocated gap", + ) + records.extend(("memory_peak", "allocator_gap")) + global_blocks = query("block") + require( + {tuple(r[k] for k in BLOCK) for r in global_blocks["rows"]} + == {tuple(r[k] for k in BLOCK) for r in blocks.values()}, + "all canonical lifetime fields/latest state", + ) + require( + len({r["lifecycle_id"] for r in global_blocks["rows"]}) == len(blocks), + "lifecycle dedup", + ) + for row in global_blocks["rows"]: + for field, event_key, action in ( + ("allocation_source", "allocEventId", 4), + ("free_source", "freeEventId", 6), + ): + event_id = row[event_key] + expected = events.get(event_id) + if expected is not None and expected["action"] == action: + source = row[field] + require( + source is not None and source["event"] == expected, + "full original source/free_completed identity", + ) + owner = next( + s["index"] + for s in oracle["slices"] + if s["startEventId"] <= event_id <= s["endEventId"] + ) + require(source["slice_index"] == owner, "cross-shard source ownership") + require( + source["frames"] is None and source["frames_status"] == "text_only", + "original no-extensions frame downgrade", + ) + else: + require(row[field] is None, "unknown source not invented") + terminal = oracle["observations"][-1] + expected_candidates = { + n + for n, b in terminal.items() + if b["state"] in (0, 1) + and b["freeEventId"] < 0 + and events.get(b["allocEventId"], {}).get("action") == 4 + } + candidates = query("leak_detection") + require( + {r["id"] for r in candidates["rows"]} == expected_candidates, + "terminal candidates, not leak proof", + ) + buckets = defaultdict(lambda: [0, 0]) + for b in blocks.values(): + if ( + events.get(b["allocEventId"], {}).get("action") == 4 + and events.get(b["freeEventId"], {}).get("action") == 6 + ): + distance = b["freeEventId"] - b["allocEventId"] + index = next( + (i for i, cap in enumerate((1000, 5000, 20000, 100000)) if distance < cap), 4 + ) + buckets[index][0] += 1 + buckets[index][1] += b["size"] + expected_lifetimes = [ + { + "lifetime_events": ("<1k", "1k-5k", "5k-20k", "20k-100k", ">=100k")[i], + "block_count": b[0], + "size_bytes": b[1], + } + for i, b in sorted(buckets.items()) + ] + require( + query("freed_block_lifetime")["rows"] == expected_lifetimes, + "completed lifetime histogram", + ) + records.extend(("block", "leak_detection", "freed_block_lifetime")) + groups = defaultdict(list) + for event in events.values(): + if event["callstack"] is not None: + groups[event["callstack"]].append(event["size"]) + expected_groups = { + (text, len(sizes), sum(sizes), sum(sizes) / len(sizes), max(sizes)) + for text, sizes in groups.items() + } + stacks = query("callstack_analysis", {"min_count": 1}) + require( + { + (r["callstack"], r["alloc_count"], r["total_size"], r["avg_size"], r["max_size"]) + for r in stacks["rows"] + } + == expected_groups, + "ALL-action stack activity, not allocation-only or live bytes", + ) + records.append("callstack_analysis") + for event_id in points: + index = next( + i + for i, s in enumerate(oracle["slices"]) + if s["startEventId"] <= event_id <= s["endEventId"] + ) + expected = { + n: b + for n, b in oracle["observations"][index].items() + if (b["allocEventId"] < 0 or b["allocEventId"] <= event_id) + and (b["freeEventId"] < 0 or b["freeEventId"] > event_id) + } + active = query("active_blocks_at_event", {"event_id": event_id}) + require({r["id"] for r in active["rows"]} == set(expected), "point active IDs") + require( + sum(r["size"] for r in active["rows"]) == sum(b["size"] for b in expected.values()), + "point active bytes", + ) + grouped = query("active_memory_callstack_at_event", {"event_id": event_id, "top_n": -1}) + require( + grouped["scope"]["source_coverage"]["active_bytes"] + == sum(b["size"] for b in expected.values()), + "unwindowed byte coverage", + ) + expected_grouped = single.execute_query( + "active_memory_callstack_at_event", + {"event_id": event_id, "top_n": -1}, + device_id=device, + )["rows"] + fields = ( + "callstack", + "category", + "size_bytes", + "requested_bytes", + "block_count", + "percent_of_active_blocks", + ) + require( + sorted(tuple(str(r[k]) for k in fields) for r in grouped["rows"]) + == sorted(tuple(str(r[k]) for k in fields) for r in expected_grouped), + "canonical standalone active attribution", + ) + preexisting = query("preexisting_live", {"event_id": event_id})["rows"][0] + pre = [ + b + for b in oracle["observations"][index].values() + if b["allocEventId"] == -1 + and ( + b["freeEventId"] is None or b["freeEventId"] > event_id or b["freeEventId"] < -1 + ) + ] + require( + preexisting == {"block_count": len(pre), "size_bytes": sum(b["size"] for b in pre)}, + "preexisting versus static occupancy", + ) + records.extend( + ("active_blocks_at_event", "active_memory_callstack_at_event", "preexisting_live") + ) + page_params = {"order_by": "active", "order_dir": "DESC", "limit": 3, "offset": 1} + page = query("event", page_params, max_rows=2) + expected_page = sorted(events.values(), key=lambda r: (-r["active"], -r["id"]))[1:3] + require( + page["rows"] == expected_page + and page["total"] == len(events) + and page["has_more"] + and page["truncated"], + "global sort/page/exact total versus coverage", + ) + overview = actual.get_database_overview() + report = ReportService() + try: + for metric in METRICS: + result = report.peak_memory_report( + root, device_id=device, metric=metric, limit=-1, timeout_s=30 + ) + event = min(events.values(), key=lambda r: (-r[metric], r["id"])) + require( + result.event_id == event["id"] and result.active_bytes_at_event == event["active"], + "report same metric event active counter", + ) + require( + result.included_bytes == sum(r["size_bytes"] for r in result.callstack_groups), + "report included byte denominator", + ) + require( + result.source_coverage["allocation_source_complete"], + "report allocation-source coverage", + ) + finally: + report.close() + require(inventory(root) == before, "all artifact bytes/members/sidecars unchanged") + return { + "status": "passed", + "templates": records, + "real_events": len(events), + "boundaries_excluded": oracle["boundaries"], + "raw_block_rows": oracle["raw_blocks"], + "canonical_lifecycles": len(blocks), + "repeated_block_rows": oracle["raw_blocks"] - len(blocks), + "terminal_candidates": len(expected_candidates), + "all_action_stack_groups": len(groups), + "points": points, + "ranges": ranges, + "overview": overview, + "hashes_before_after": before, + "reference_memory": "small materialized oracle; NOT product memory-bound evidence", + "GUI": "human-deferred/pending/not-run", + "original_producer": "not-run by this runner", + } + + +def compare_forward(original: Path, compatible: Path, device: int) -> dict: + """Physical forward-export comparison; NOT actual C++/GUI consumption.""" + before = (inventory(original), inventory(compatible)) + expected = physical_oracle(original, device) + actual = physical_oracle(compatible, device) + for field in ( + "events", + "dictionaries", + "schemas", + "boundaries", + "raw_blocks", + ): + require(actual[field] == expected[field], "forward physical " + field) + + def partition(rows): + positive = {n: row for n, row in rows.items() if n >= 0} + negative = Counter( + tuple(row[k] for k in BLOCK if k != "id") for n, row in rows.items() if n < 0 + ) + return positive, negative + + require( + partition(actual["blocks"]) == partition(expected["blocks"]), + "forward physical blocks: exact real IDs; negative field MULTISET", + ) + require( + len(actual["observations"]) == len(expected["observations"]), "forward physical shard count" + ) + for index, (own, reference) in enumerate( + zip(actual["observations"], expected["observations"], strict=True) + ): + require( + partition(own) == partition(reference), + f"forward physical shard {index}: exact real IDs; negative field MULTISET", + ) + require( + before == (inventory(original), inventory(compatible)), "forward all-member preservation" + ) + return { + "status": "passed", + "scope": "physical export versus existing original artifact; GUI/C++ consumer pending", + "negative_identity": "producer-local tokens: cross-producer identity UNPROVED; exact six-field/state multisets global and per shard; own-token stability separately validated; no address dedup or ID rewriting", + "real_events": len(actual["events"]), + "raw_block_rows": actual["raw_blocks"], + "all_member_hashes": list(before), + } + + +def cli_acceptance(root: Path, device: int, output: Path) -> list[dict]: + """Write ONLY the new evidence directory's test focus; input remains read-only.""" + import pt_snap_cli.api as source_api + from pt_snap_cli.api import SnapshotAnalyzer + + home = output / "isolated-home" + home.mkdir() + env = {**os.environ, "HOME": str(home), "PYTHONDONTWRITEBYTECODE": "1"} + env.pop("PT_SNAP_DB_PATH", None) + parent_api = Path(source_api.__file__).resolve() + inherited = [ + str(Path(part or ".").absolute()) for part in env.get("PYTHONPATH", "").split(os.pathsep) + ] + env["PYTHONPATH"] = os.pathsep.join([str(parent_api.parents[1]), *inherited]) + probe_command = [ + sys.executable, + "-c", + "import pathlib,pt_snap_cli.api; print(pathlib.Path(pt_snap_cli.api.__file__).resolve())", + ] + probe = subprocess.run( + probe_command, cwd=output, env=env, capture_output=True, text=True, timeout=30 + ) + require( + probe.returncode == 0 and probe.stdout.strip() == str(parent_api), + "child uses the exact parent API source", + ) + commands = [ + ["focus", str(root), "--device", str(device), "--json"], + [ + "query", + "--template-use", + "event", + "--params", + '{"order_by":"active","order_dir":"DESC","limit":3,"offset":1}', + "-n", + "2", + "--exact-total", + "--json", + ], + [ + "report", + "peak-memory", + str(root), + "--device", + str(device), + "--metric", + "reserved", + "--limit", + "-1", + "--json", + ], + ] + records = [] + for args in commands: + command = [sys.executable, "-m", "pt_snap_cli.cli", *args] + result = subprocess.run( + command, cwd=output, env=env, capture_output=True, text=True, timeout=60 + ) + records.append( + { + "command": command, + "cwd": str(output), + "source_api": str(parent_api), + "source_identity_probe": { + "command": probe_command, + "cwd": str(output), + "exit": probe.returncode, + "stdout": probe.stdout, + "stderr": probe.stderr, + }, + "env": { + key: env[key] + for key in ("HOME", "PYTHONDONTWRITEBYTECODE", "PYTHONPATH") + if key in env + }, + "exit": result.returncode, + "stderr": result.stderr, + "status": "passed" if result.returncode == 0 else "failed", + } + ) + require(result.returncode == 0, "actual CLI " + args[0]) + data = json.loads(result.stdout) + if args[0] == "query": + with SnapshotAnalyzer(root) as analyzer: + api = analyzer.execute_query( + "event", + {"order_by": "active", "order_dir": "DESC", "limit": 3, "offset": 1}, + device_id=device, + max_rows=2, + exact_total=True, + ) + require( + all( + data[k] == api[k] + for k in ( + "rows", + "total", + "returned", + "scope", + "has_more", + "truncated", + "total_is_exact", + ) + ), + "actual CLI/API normalized scope/page", + ) + if args[0] == "report": + require( + data["source_coverage"]["allocation_source_complete"] + and data["scope"]["range_complete"], + "actual CLI report coverage", + ) + records[-1]["results"] = { + k: data[k] + for k in ( + "event_id", + "returned", + "total", + "scope", + "has_more", + "truncated", + "source_coverage", + ) + if k in data + } + return records + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--artifact", required=True, type=Path) + parser.add_argument("--receipt", required=True, type=Path) + parser.add_argument( + "--source", type=Path, help="Optional hash-only original pickle identity check" + ) + parser.add_argument( + "--compatible", + type=Path, + help="Optional EXISTING pt-snap physical export; never generated here", + ) + parser.add_argument("--device", required=True, type=int) + parser.add_argument( + "--output", required=True, type=Path, help="NEW exclusive evidence directory" + ) + args = parser.parse_args() + receipt = read_json(args.receipt) + root = canonical(args.artifact) + verify_receipt(root, receipt, args.source) # before any SQLite/product open + output = canonical(args.output) + disjoint_output(root, output) + if args.compatible is not None: + compatible = canonical(args.compatible) + disjoint_output(compatible, output) + manifest = read_json(compatible / "manifest.json") + identity = manifest.get("ptSnap", {}).get("identity", {}) + require( + identity.get("sourceSha256") == receipt.get("fixtureSHA256") + and manifest.get("cacheHash") == receipt.get("saltedCacheHash"), + "forward same source identity", + ) + if type(args.device) is not int or args.device < 0: + raise ValueError("Explicit nonnegative device required") + output.mkdir() # never adopt/replace an existing directory + try: + result = compare_artifact(root, args.device, output / "canonical-reference.db") + if args.compatible is not None: + result["forward_physical"] = compare_forward(root, compatible, args.device) + result["cli_acceptance"] = cli_acceptance(root, args.device, output) + verify_receipt(root, receipt, args.source) + except Exception as exc: + result = {"status": "failed", "error": repr(exc), "GUI": "pending/not-run"} + with (output / "acceptance.json").open("x") as file: + json.dump(result, file, indent=2) + raise + result.update( + { + "revision": REVISION, + "receipt_authority": "explicit caller-reviewed input; authenticity not established by hashes", + "command": sys.argv, + "cwd": str(Path.cwd()), + "python": sys.executable, + } + ) + with (output / "acceptance.json").open("x") as file: + json.dump(result, file, indent=2) + print( + json.dumps( + { + "status": result["status"], + "real_events": result["real_events"], + "output": str(output), + } + ) + ) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/docs/README.md b/docs/README.md index 619d4f1..7ee95a6 100644 --- a/docs/README.md +++ b/docs/README.md @@ -12,6 +12,8 @@ | Splitting Snapshots | [en/splitting.md](en/splitting.md) | | Agent skills | [en/skills.md](en/skills.md) | | Database Schema | [en/database.md](en/database.md) | +| Sharded protocols, native/compatible imports and dataset queries | [en/sharded-snapshotdb.md](en/sharded-snapshotdb.md) | +| Interoperability acceptance, performance and pending GUI checklist | [en/interop-acceptance.md](en/interop-acceptance.md) | | Import metadata | [en/database.md](en/database.md) | | SnapshotAnalyzer API | [en/snapshot-analyzer-api.md](en/snapshot-analyzer-api.md) | | ResultMapper API | [en/result-mapper-api.md](en/result-mapper-api.md) | @@ -28,6 +30,8 @@ | 拆分快照 | [zh/splitting.md](zh/splitting.md) | | Agent skill | [zh/skills.md](zh/skills.md) | | 数据库格式 | [zh/database.md](zh/database.md) | +| 分片协议、原生/兼容导入与数据集查询 | [zh/sharded-snapshotdb.md](zh/sharded-snapshotdb.md) | +| 双向互操作验收、性能与待验收 GUI 清单 | [zh/interop-acceptance.md](zh/interop-acceptance.md) | | 导入 metadata | [zh/database.md](zh/database.md) | | SnapshotAnalyzer API | [zh/snapshot-analyzer-api.md](zh/snapshot-analyzer-api.md) | | ResultMapper API | [zh/result-mapper-api.md](zh/result-mapper-api.md) | diff --git a/docs/dataset-performance-baseline.json b/docs/dataset-performance-baseline.json new file mode 100644 index 0000000..d8a6dd0 --- /dev/null +++ b/docs/dataset-performance-baseline.json @@ -0,0 +1,1487 @@ +{ + "status": "passed", + "source_head": "a0f9738fedaa31b7c3929f5884c48df74616a75a", + "benchmark_script_sha256": "100a04614c174b87a7365e298c0d85323be7ee3f4986a56f95ef0ddc5ec1c6f1", + "cold_definition": "fresh output/process, no warmup; OS page cache NOT cleared", + "reuse_definition": "fresh process reusing only this run's recognized owned artifact, force=False", + "original_producer": "not-run; preexisting single cold wall is not comparable RSS/phase evidence", + "GUI": "pending/not-run", + "limits": [ + "Pickle loads in full memory; sharding is not streaming", + "Shards repeat long-lived blocks; compatible format repeats inline stack text", + "No unstable elapsed-time assertion or performance benefit promise", + "Query latency includes validation/member hashing and repeated scans; only one local observation" + ], + "schema_version": 1, + "evidence_scope": "one bounded local same-condition pt-snap observation per mode/phase; no speed assertions or comparative original-producer result", + "source_fixture": { + "name": "snapshot_expandable.pkl", + "bytes": 581140, + "sha256": "3afc9d1c5ef4ca4b417e58c0830e9eb8a913eb9459f4088b8c66c22325c68c40" + }, + "command_shape": [ + "python", + "benchmarks/dataset_import_baseline.py", + "--source", + "tests/fixtures/snapshots/snapshot_expandable.pkl", + "--sha256", + "3afc9d1c5ef4ca4b417e58c0830e9eb8a913eb9459f4088b8c66c22325c68c40", + "--device", + "0", + "--capacity", + "2000", + "--output", + "", + "--trusted-pickle" + ], + "command_scope": "reproducible shape, not a claim that placeholder paths were executed; exact actual commands/cwd/env/exits/logs retained in local performance.json and durable issue ledger", + "records": [ + { + "mode": "single-db", + "phase": "cold", + "status": "passed", + "import_wall_seconds": 0.05691691697575152, + "isolated_import_process_peak_rss_raw": 48152576, + "isolated_import_process_peak_rss_kib": 47024.0, + "isolated_import_process_peak_rss_mib": 45.921875, + "rss_scope": "RUSAGE_SELF fresh child high-water through import, before queries; includes interpreter/imports", + "phase_timings": { + "load_representation": { + "seconds": 0.002031833049841225, + "calls": 1 + }, + "standalone_replay_with_sqlite_hooks": { + "seconds": 0.029698416008614004, + "calls": 1 + } + }, + "phase_timing_scope": "named wrapped functions, inclusive; sharded replay includes SQLite writes/backfills, NOT parser-only", + "phase_rss": { + "status": "unavailable", + "reason": "No per-phase sampler/isolation; cumulative high-waters cannot be subtracted into load/replay peaks" + }, + "other_phase_timings": { + "status": "unavailable", + "reason": "validation/hash/metadata/publication remainder not separately instrumented" + }, + "reused": false, + "device": 0, + "capacity": null, + "total_disk_bytes": 1601536, + "individual_files_bytes": { + "snapshot_expandable.pkl.db": 1601536 + }, + "individual_shard_bytes": { + "snapshot_expandable.pkl.db": 1601536 + }, + "raw_block_rows": 3219, + "unique_block_invariants": 3219, + "repeated_block_rows": 0, + "repeated_block_cost": { + "repeated_base_integer_cells": 0, + "physical_disk_attribution": "unavailable: shared SQLite pages/indexes cannot be apportioned to duplicated rows by file-size subtraction" + }, + "errors": [], + "python_version": "3.13.11 | packaged by Anaconda, Inc. | (main, Dec 10 2025, 21:21:08) [Clang 20.1.8 ]", + "platform": "macOS-26.0.1-arm64-arm-64bit-Mach-O", + "exit": 0, + "queries": { + "active_blocks_at_event": { + "seconds": 0.0037044170312583447, + "returned": 645, + "total": 645, + "has_more": false, + "truncated": false, + "status": "passed", + "dataset_support": { + "supported": true, + "semantics": "owner_event_active_set_cross_slice_sources", + "device_scope": "one_selected_device; never sum device peaks", + "custom_sql": "unsupported; select an explicit standalone member", + "work_limit_rows": 100000, + "work_limit_bytes": 67108864, + "budget_scope": "one_query_or_explicit_report_composition" + } + }, + "active_memory_callstack_at_event": { + "seconds": 0.00290229101665318, + "returned": 4, + "total": 4, + "has_more": false, + "truncated": false, + "status": "passed", + "dataset_support": { + "supported": true, + "semantics": "owner_event_global_source_groups_before_top_n", + "device_scope": "one_selected_device; never sum device peaks", + "custom_sql": "unsupported; select an explicit standalone member", + "work_limit_rows": 100000, + "work_limit_bytes": 67108864, + "budget_scope": "one_query_or_explicit_report_composition" + } + }, + "allocation": { + "seconds": 0.005575167015194893, + "returned": 8094, + "total": 8094, + "has_more": false, + "truncated": false, + "status": "passed", + "dataset_support": { + "supported": true, + "semantics": "real_events_global_filter_sort_page", + "device_scope": "one_selected_device; never sum device peaks", + "custom_sql": "unsupported; select an explicit standalone member", + "work_limit_rows": 100000, + "work_limit_bytes": 67108864, + "budget_scope": "one_query_or_explicit_report_composition" + } + }, + "allocator_gap": { + "seconds": 0.0037931660190224648, + "returned": 1, + "total": 1, + "has_more": false, + "truncated": false, + "status": "passed", + "dataset_support": { + "supported": true, + "semantics": "global_peaks_same_event_counters", + "device_scope": "one_selected_device; never sum device peaks", + "custom_sql": "unsupported; select an explicit standalone member", + "work_limit_rows": 100000, + "work_limit_bytes": 67108864, + "budget_scope": "one_query_or_explicit_report_composition" + } + }, + "block": { + "seconds": 0.004353124997578561, + "returned": 3219, + "total": 3219, + "has_more": false, + "truncated": false, + "status": "passed", + "dataset_support": { + "supported": true, + "semantics": "proved_lifecycles_dedup_latest_observation", + "device_scope": "one_selected_device; never sum device peaks", + "custom_sql": "unsupported; select an explicit standalone member", + "work_limit_rows": 100000, + "work_limit_bytes": 67108864, + "budget_scope": "one_query_or_explicit_report_composition" + } + }, + "callstack_analysis": { + "seconds": 0.018826082930900156, + "returned": 60, + "total": 60, + "has_more": false, + "truncated": false, + "status": "passed", + "dataset_support": { + "supported": true, + "semantics": "all_real_stack_bearing_actions_global_identity_weighted_stats", + "device_scope": "one_selected_device; never sum device peaks", + "custom_sql": "unsupported; select an explicit standalone member", + "work_limit_rows": 100000, + "work_limit_bytes": 67108864, + "budget_scope": "one_query_or_explicit_report_composition" + } + }, + "event": { + "seconds": 0.001940542017109692, + "returned": 1, + "total": 1, + "has_more": false, + "truncated": false, + "status": "passed", + "dataset_support": { + "supported": true, + "semantics": "real_events_global_filter_sort_page", + "device_scope": "one_selected_device; never sum device peaks", + "custom_sql": "unsupported; select an explicit standalone member", + "work_limit_rows": 100000, + "work_limit_bytes": 67108864, + "budget_scope": "one_query_or_explicit_report_composition" + } + }, + "freed_block_lifetime": { + "seconds": 0.001361750066280365, + "returned": 2, + "total": 2, + "has_more": false, + "truncated": false, + "status": "passed", + "dataset_support": { + "supported": true, + "semantics": "proved_completed_lifecycles_once", + "device_scope": "one_selected_device; never sum device peaks", + "custom_sql": "unsupported; select an explicit standalone member", + "work_limit_rows": 100000, + "work_limit_bytes": 67108864, + "budget_scope": "one_query_or_explicit_report_composition" + } + }, + "leak_detection": { + "seconds": 0.001192792085930705, + "returned": 325, + "total": 325, + "has_more": false, + "truncated": false, + "status": "passed", + "dataset_support": { + "supported": true, + "semantics": "terminal_dataset_dynamic_survivors_not_confirmed_leaks", + "device_scope": "one_selected_device; never sum device peaks", + "custom_sql": "unsupported; select an explicit standalone member", + "work_limit_rows": 100000, + "work_limit_bytes": 67108864, + "budget_scope": "one_query_or_explicit_report_composition" + } + }, + "memory_peak": { + "seconds": 0.0010200829710811377, + "returned": 1, + "total": 1, + "has_more": false, + "truncated": false, + "status": "passed", + "dataset_support": { + "supported": true, + "semantics": "independent_real_counter_peaks_earliest_id", + "device_scope": "one_selected_device; never sum device peaks", + "custom_sql": "unsupported; select an explicit standalone member", + "work_limit_rows": 100000, + "work_limit_bytes": 67108864, + "budget_scope": "one_query_or_explicit_report_composition" + } + }, + "preexisting_live": { + "seconds": 0.0003534170100465417, + "returned": 1, + "total": 1, + "has_more": false, + "truncated": false, + "status": "passed", + "dataset_support": { + "supported": true, + "semantics": "owner_event_preexisting_occupancy", + "device_scope": "one_selected_device; never sum device peaks", + "custom_sql": "unsupported; select an explicit standalone member", + "work_limit_rows": 100000, + "work_limit_bytes": 67108864, + "budget_scope": "one_query_or_explicit_report_composition" + } + } + } + }, + { + "mode": "single-db", + "phase": "reuse", + "status": "passed", + "import_wall_seconds": 0.0008004589471966028, + "isolated_import_process_peak_rss_raw": 34799616, + "isolated_import_process_peak_rss_kib": 33984.0, + "isolated_import_process_peak_rss_mib": 33.1875, + "rss_scope": "RUSAGE_SELF fresh child high-water through import, before queries; includes interpreter/imports", + "phase_timings": {}, + "phase_timing_scope": "named wrapped functions, inclusive; sharded replay includes SQLite writes/backfills, NOT parser-only", + "phase_rss": { + "status": "unavailable", + "reason": "No per-phase sampler/isolation; cumulative high-waters cannot be subtracted into load/replay peaks" + }, + "other_phase_timings": { + "status": "unavailable", + "reason": "validation/hash/metadata/publication remainder not separately instrumented" + }, + "reused": true, + "device": 0, + "capacity": null, + "total_disk_bytes": 1601536, + "individual_files_bytes": { + "snapshot_expandable.pkl.db": 1601536 + }, + "individual_shard_bytes": { + "snapshot_expandable.pkl.db": 1601536 + }, + "raw_block_rows": 3219, + "unique_block_invariants": 3219, + "repeated_block_rows": 0, + "repeated_block_cost": { + "repeated_base_integer_cells": 0, + "physical_disk_attribution": "unavailable: shared SQLite pages/indexes cannot be apportioned to duplicated rows by file-size subtraction" + }, + "errors": [], + "python_version": "3.13.11 | packaged by Anaconda, Inc. | (main, Dec 10 2025, 21:21:08) [Clang 20.1.8 ]", + "platform": "macOS-26.0.1-arm64-arm-64bit-Mach-O", + "exit": 0, + "queries": { + "active_blocks_at_event": { + "seconds": 0.0036053750663995743, + "returned": 645, + "total": 645, + "has_more": false, + "truncated": false, + "status": "passed", + "dataset_support": { + "supported": true, + "semantics": "owner_event_active_set_cross_slice_sources", + "device_scope": "one_selected_device; never sum device peaks", + "custom_sql": "unsupported; select an explicit standalone member", + "work_limit_rows": 100000, + "work_limit_bytes": 67108864, + "budget_scope": "one_query_or_explicit_report_composition" + } + }, + "active_memory_callstack_at_event": { + "seconds": 0.002839708002284169, + "returned": 4, + "total": 4, + "has_more": false, + "truncated": false, + "status": "passed", + "dataset_support": { + "supported": true, + "semantics": "owner_event_global_source_groups_before_top_n", + "device_scope": "one_selected_device; never sum device peaks", + "custom_sql": "unsupported; select an explicit standalone member", + "work_limit_rows": 100000, + "work_limit_bytes": 67108864, + "budget_scope": "one_query_or_explicit_report_composition" + } + }, + "allocation": { + "seconds": 0.005784249980933964, + "returned": 8094, + "total": 8094, + "has_more": false, + "truncated": false, + "status": "passed", + "dataset_support": { + "supported": true, + "semantics": "real_events_global_filter_sort_page", + "device_scope": "one_selected_device; never sum device peaks", + "custom_sql": "unsupported; select an explicit standalone member", + "work_limit_rows": 100000, + "work_limit_bytes": 67108864, + "budget_scope": "one_query_or_explicit_report_composition" + } + }, + "allocator_gap": { + "seconds": 0.0036612499970942736, + "returned": 1, + "total": 1, + "has_more": false, + "truncated": false, + "status": "passed", + "dataset_support": { + "supported": true, + "semantics": "global_peaks_same_event_counters", + "device_scope": "one_selected_device; never sum device peaks", + "custom_sql": "unsupported; select an explicit standalone member", + "work_limit_rows": 100000, + "work_limit_bytes": 67108864, + "budget_scope": "one_query_or_explicit_report_composition" + } + }, + "block": { + "seconds": 0.004391541937366128, + "returned": 3219, + "total": 3219, + "has_more": false, + "truncated": false, + "status": "passed", + "dataset_support": { + "supported": true, + "semantics": "proved_lifecycles_dedup_latest_observation", + "device_scope": "one_selected_device; never sum device peaks", + "custom_sql": "unsupported; select an explicit standalone member", + "work_limit_rows": 100000, + "work_limit_bytes": 67108864, + "budget_scope": "one_query_or_explicit_report_composition" + } + }, + "callstack_analysis": { + "seconds": 0.018302790937013924, + "returned": 60, + "total": 60, + "has_more": false, + "truncated": false, + "status": "passed", + "dataset_support": { + "supported": true, + "semantics": "all_real_stack_bearing_actions_global_identity_weighted_stats", + "device_scope": "one_selected_device; never sum device peaks", + "custom_sql": "unsupported; select an explicit standalone member", + "work_limit_rows": 100000, + "work_limit_bytes": 67108864, + "budget_scope": "one_query_or_explicit_report_composition" + } + }, + "event": { + "seconds": 0.0019014590652659535, + "returned": 1, + "total": 1, + "has_more": false, + "truncated": false, + "status": "passed", + "dataset_support": { + "supported": true, + "semantics": "real_events_global_filter_sort_page", + "device_scope": "one_selected_device; never sum device peaks", + "custom_sql": "unsupported; select an explicit standalone member", + "work_limit_rows": 100000, + "work_limit_bytes": 67108864, + "budget_scope": "one_query_or_explicit_report_composition" + } + }, + "freed_block_lifetime": { + "seconds": 0.0013899169862270355, + "returned": 2, + "total": 2, + "has_more": false, + "truncated": false, + "status": "passed", + "dataset_support": { + "supported": true, + "semantics": "proved_completed_lifecycles_once", + "device_scope": "one_selected_device; never sum device peaks", + "custom_sql": "unsupported; select an explicit standalone member", + "work_limit_rows": 100000, + "work_limit_bytes": 67108864, + "budget_scope": "one_query_or_explicit_report_composition" + } + }, + "leak_detection": { + "seconds": 0.0012190000852569938, + "returned": 325, + "total": 325, + "has_more": false, + "truncated": false, + "status": "passed", + "dataset_support": { + "supported": true, + "semantics": "terminal_dataset_dynamic_survivors_not_confirmed_leaks", + "device_scope": "one_selected_device; never sum device peaks", + "custom_sql": "unsupported; select an explicit standalone member", + "work_limit_rows": 100000, + "work_limit_bytes": 67108864, + "budget_scope": "one_query_or_explicit_report_composition" + } + }, + "memory_peak": { + "seconds": 0.0010333340615034103, + "returned": 1, + "total": 1, + "has_more": false, + "truncated": false, + "status": "passed", + "dataset_support": { + "supported": true, + "semantics": "independent_real_counter_peaks_earliest_id", + "device_scope": "one_selected_device; never sum device peaks", + "custom_sql": "unsupported; select an explicit standalone member", + "work_limit_rows": 100000, + "work_limit_bytes": 67108864, + "budget_scope": "one_query_or_explicit_report_composition" + } + }, + "preexisting_live": { + "seconds": 0.0003636250039562583, + "returned": 1, + "total": 1, + "has_more": false, + "truncated": false, + "status": "passed", + "dataset_support": { + "supported": true, + "semantics": "owner_event_preexisting_occupancy", + "device_scope": "one_selected_device; never sum device peaks", + "custom_sql": "unsupported; select an explicit standalone member", + "work_limit_rows": 100000, + "work_limit_bytes": 67108864, + "budget_scope": "one_query_or_explicit_report_composition" + } + } + } + }, + { + "mode": "pt-snap-native-v2", + "phase": "cold", + "status": "passed", + "import_wall_seconds": 0.15482937498018146, + "isolated_import_process_peak_rss_raw": 61325312, + "isolated_import_process_peak_rss_kib": 59888.0, + "isolated_import_process_peak_rss_mib": 58.484375, + "rss_scope": "RUSAGE_SELF fresh child high-water through import, before queries; includes interpreter/imports", + "phase_timings": { + "load_representation": { + "seconds": 0.0019707499304786325, + "calls": 1 + }, + "sharded_replay_with_sqlite_and_backfill": { + "seconds": 0.11052991601172835, + "calls": 1 + } + }, + "phase_timing_scope": "named wrapped functions, inclusive; sharded replay includes SQLite writes/backfills, NOT parser-only", + "phase_rss": { + "status": "unavailable", + "reason": "No per-phase sampler/isolation; cumulative high-waters cannot be subtracted into load/replay peaks" + }, + "other_phase_timings": { + "status": "unavailable", + "reason": "validation/hash/metadata/publication remainder not separately instrumented" + }, + "reused": false, + "device": 0, + "capacity": 2000, + "total_disk_bytes": 9923330, + "individual_files_bytes": { + "snapshot_expandable.pkl.pt-snap-native-v2/manifest.json": 2818, + "snapshot_expandable.pkl.pt-snap-native-v2/device_0/slice_00004.db": 442368, + "snapshot_expandable.pkl.pt-snap-native-v2/device_0/slice_00000.db": 1875968, + "snapshot_expandable.pkl.pt-snap-native-v2/device_0/slice_00001.db": 2605056, + "snapshot_expandable.pkl.pt-snap-native-v2/device_0/slice_00002.db": 2772992, + "snapshot_expandable.pkl.pt-snap-native-v2/device_0/slice_00003.db": 2224128 + }, + "individual_shard_bytes": { + "snapshot_expandable.pkl.pt-snap-native-v2/device_0/slice_00000.db": 1875968, + "snapshot_expandable.pkl.pt-snap-native-v2/device_0/slice_00001.db": 2605056, + "snapshot_expandable.pkl.pt-snap-native-v2/device_0/slice_00002.db": 2772992, + "snapshot_expandable.pkl.pt-snap-native-v2/device_0/slice_00003.db": 2224128, + "snapshot_expandable.pkl.pt-snap-native-v2/device_0/slice_00004.db": 442368 + }, + "raw_block_rows": 6412, + "unique_block_invariants": 3219, + "repeated_block_rows": 3193, + "repeated_block_cost": { + "repeated_base_integer_cells": 22351, + "physical_disk_attribution": "unavailable: shared SQLite pages/indexes cannot be apportioned to duplicated rows by file-size subtraction" + }, + "errors": [], + "python_version": "3.13.11 | packaged by Anaconda, Inc. | (main, Dec 10 2025, 21:21:08) [Clang 20.1.8 ]", + "platform": "macOS-26.0.1-arm64-arm-64bit-Mach-O", + "exit": 0, + "queries": { + "active_blocks_at_event": { + "seconds": 0.04044750006869435, + "returned": 645, + "total": 645, + "has_more": false, + "truncated": false, + "status": "passed", + "dataset_support": { + "supported": true, + "semantics": "owner_event_active_set_cross_slice_sources", + "device_scope": "one_selected_device; never sum device peaks", + "custom_sql": "unsupported; select an explicit standalone member", + "work_limit_rows": 100000, + "work_limit_bytes": 67108864, + "budget_scope": "one_query_or_explicit_report_composition" + } + }, + "active_memory_callstack_at_event": { + "seconds": 0.0398883749730885, + "returned": 4, + "total": 4, + "has_more": false, + "truncated": false, + "status": "passed", + "dataset_support": { + "supported": true, + "semantics": "owner_event_global_source_groups_before_top_n", + "device_scope": "one_selected_device; never sum device peaks", + "custom_sql": "unsupported; select an explicit standalone member", + "work_limit_rows": 100000, + "work_limit_bytes": 67108864, + "budget_scope": "one_query_or_explicit_report_composition" + } + }, + "allocation": { + "seconds": 0.06903308292385191, + "returned": 8092, + "total": 8092, + "has_more": false, + "truncated": false, + "status": "passed", + "dataset_support": { + "supported": true, + "semantics": "real_events_global_filter_sort_page", + "device_scope": "one_selected_device; never sum device peaks", + "custom_sql": "unsupported; select an explicit standalone member", + "work_limit_rows": 100000, + "work_limit_bytes": 67108864, + "budget_scope": "one_query_or_explicit_report_composition" + } + }, + "allocator_gap": { + "seconds": 0.057917125057429075, + "returned": 1, + "total": 1, + "has_more": false, + "truncated": false, + "status": "passed", + "dataset_support": { + "supported": true, + "semantics": "global_peaks_same_event_counters", + "device_scope": "one_selected_device; never sum device peaks", + "custom_sql": "unsupported; select an explicit standalone member", + "work_limit_rows": 100000, + "work_limit_bytes": 67108864, + "budget_scope": "one_query_or_explicit_report_composition" + } + }, + "block": { + "seconds": 0.1301991669461131, + "returned": 3219, + "total": 3219, + "has_more": false, + "truncated": false, + "status": "passed", + "dataset_support": { + "supported": true, + "semantics": "proved_lifecycles_dedup_latest_observation", + "device_scope": "one_selected_device; never sum device peaks", + "custom_sql": "unsupported; select an explicit standalone member", + "work_limit_rows": 100000, + "work_limit_bytes": 67108864, + "budget_scope": "one_query_or_explicit_report_composition" + } + }, + "callstack_analysis": { + "seconds": 0.10538816708140075, + "returned": 60, + "total": 60, + "has_more": false, + "truncated": false, + "status": "passed", + "dataset_support": { + "supported": true, + "semantics": "all_real_stack_bearing_actions_global_identity_weighted_stats", + "device_scope": "one_selected_device; never sum device peaks", + "custom_sql": "unsupported; select an explicit standalone member", + "work_limit_rows": 100000, + "work_limit_bytes": 67108864, + "budget_scope": "one_query_or_explicit_report_composition" + } + }, + "event": { + "seconds": 0.034352708025835454, + "returned": 1, + "total": 1, + "has_more": false, + "truncated": false, + "status": "passed", + "dataset_support": { + "supported": true, + "semantics": "real_events_global_filter_sort_page", + "device_scope": "one_selected_device; never sum device peaks", + "custom_sql": "unsupported; select an explicit standalone member", + "work_limit_rows": 100000, + "work_limit_bytes": 67108864, + "budget_scope": "one_query_or_explicit_report_composition" + } + }, + "freed_block_lifetime": { + "seconds": 0.11085683398414403, + "returned": 2, + "total": 2, + "has_more": false, + "truncated": false, + "status": "passed", + "dataset_support": { + "supported": true, + "semantics": "proved_completed_lifecycles_once", + "device_scope": "one_selected_device; never sum device peaks", + "custom_sql": "unsupported; select an explicit standalone member", + "work_limit_rows": 100000, + "work_limit_bytes": 67108864, + "budget_scope": "one_query_or_explicit_report_composition" + } + }, + "leak_detection": { + "seconds": 0.10605162498541176, + "returned": 325, + "total": 325, + "has_more": false, + "truncated": false, + "status": "passed", + "dataset_support": { + "supported": true, + "semantics": "terminal_dataset_dynamic_survivors_not_confirmed_leaks", + "device_scope": "one_selected_device; never sum device peaks", + "custom_sql": "unsupported; select an explicit standalone member", + "work_limit_rows": 100000, + "work_limit_bytes": 67108864, + "budget_scope": "one_query_or_explicit_report_composition" + } + }, + "memory_peak": { + "seconds": 0.05739520804490894, + "returned": 1, + "total": 1, + "has_more": false, + "truncated": false, + "status": "passed", + "dataset_support": { + "supported": true, + "semantics": "independent_real_counter_peaks_earliest_id", + "device_scope": "one_selected_device; never sum device peaks", + "custom_sql": "unsupported; select an explicit standalone member", + "work_limit_rows": 100000, + "work_limit_bytes": 67108864, + "budget_scope": "one_query_or_explicit_report_composition" + } + }, + "preexisting_live": { + "seconds": 0.03284445800818503, + "returned": 1, + "total": 1, + "has_more": false, + "truncated": false, + "status": "passed", + "dataset_support": { + "supported": true, + "semantics": "owner_event_preexisting_occupancy", + "device_scope": "one_selected_device; never sum device peaks", + "custom_sql": "unsupported; select an explicit standalone member", + "work_limit_rows": 100000, + "work_limit_bytes": 67108864, + "budget_scope": "one_query_or_explicit_report_composition" + } + } + } + }, + { + "mode": "pt-snap-native-v2", + "phase": "reuse", + "status": "passed", + "import_wall_seconds": 0.03841129201464355, + "isolated_import_process_peak_rss_raw": 46514176, + "isolated_import_process_peak_rss_kib": 45424.0, + "isolated_import_process_peak_rss_mib": 44.359375, + "rss_scope": "RUSAGE_SELF fresh child high-water through import, before queries; includes interpreter/imports", + "phase_timings": {}, + "phase_timing_scope": "named wrapped functions, inclusive; sharded replay includes SQLite writes/backfills, NOT parser-only", + "phase_rss": { + "status": "unavailable", + "reason": "No per-phase sampler/isolation; cumulative high-waters cannot be subtracted into load/replay peaks" + }, + "other_phase_timings": { + "status": "unavailable", + "reason": "validation/hash/metadata/publication remainder not separately instrumented" + }, + "reused": true, + "device": 0, + "capacity": 2000, + "total_disk_bytes": 9923330, + "individual_files_bytes": { + "snapshot_expandable.pkl.pt-snap-native-v2/manifest.json": 2818, + "snapshot_expandable.pkl.pt-snap-native-v2/device_0/slice_00004.db": 442368, + "snapshot_expandable.pkl.pt-snap-native-v2/device_0/slice_00000.db": 1875968, + "snapshot_expandable.pkl.pt-snap-native-v2/device_0/slice_00001.db": 2605056, + "snapshot_expandable.pkl.pt-snap-native-v2/device_0/slice_00002.db": 2772992, + "snapshot_expandable.pkl.pt-snap-native-v2/device_0/slice_00003.db": 2224128 + }, + "individual_shard_bytes": { + "snapshot_expandable.pkl.pt-snap-native-v2/device_0/slice_00000.db": 1875968, + "snapshot_expandable.pkl.pt-snap-native-v2/device_0/slice_00001.db": 2605056, + "snapshot_expandable.pkl.pt-snap-native-v2/device_0/slice_00002.db": 2772992, + "snapshot_expandable.pkl.pt-snap-native-v2/device_0/slice_00003.db": 2224128, + "snapshot_expandable.pkl.pt-snap-native-v2/device_0/slice_00004.db": 442368 + }, + "raw_block_rows": 6412, + "unique_block_invariants": 3219, + "repeated_block_rows": 3193, + "repeated_block_cost": { + "repeated_base_integer_cells": 22351, + "physical_disk_attribution": "unavailable: shared SQLite pages/indexes cannot be apportioned to duplicated rows by file-size subtraction" + }, + "errors": [], + "python_version": "3.13.11 | packaged by Anaconda, Inc. | (main, Dec 10 2025, 21:21:08) [Clang 20.1.8 ]", + "platform": "macOS-26.0.1-arm64-arm-64bit-Mach-O", + "exit": 0, + "queries": { + "active_blocks_at_event": { + "seconds": 0.04047504207119346, + "returned": 645, + "total": 645, + "has_more": false, + "truncated": false, + "status": "passed", + "dataset_support": { + "supported": true, + "semantics": "owner_event_active_set_cross_slice_sources", + "device_scope": "one_selected_device; never sum device peaks", + "custom_sql": "unsupported; select an explicit standalone member", + "work_limit_rows": 100000, + "work_limit_bytes": 67108864, + "budget_scope": "one_query_or_explicit_report_composition" + } + }, + "active_memory_callstack_at_event": { + "seconds": 0.03824825002811849, + "returned": 4, + "total": 4, + "has_more": false, + "truncated": false, + "status": "passed", + "dataset_support": { + "supported": true, + "semantics": "owner_event_global_source_groups_before_top_n", + "device_scope": "one_selected_device; never sum device peaks", + "custom_sql": "unsupported; select an explicit standalone member", + "work_limit_rows": 100000, + "work_limit_bytes": 67108864, + "budget_scope": "one_query_or_explicit_report_composition" + } + }, + "allocation": { + "seconds": 0.0668305829167366, + "returned": 8092, + "total": 8092, + "has_more": false, + "truncated": false, + "status": "passed", + "dataset_support": { + "supported": true, + "semantics": "real_events_global_filter_sort_page", + "device_scope": "one_selected_device; never sum device peaks", + "custom_sql": "unsupported; select an explicit standalone member", + "work_limit_rows": 100000, + "work_limit_bytes": 67108864, + "budget_scope": "one_query_or_explicit_report_composition" + } + }, + "allocator_gap": { + "seconds": 0.057496541063301265, + "returned": 1, + "total": 1, + "has_more": false, + "truncated": false, + "status": "passed", + "dataset_support": { + "supported": true, + "semantics": "global_peaks_same_event_counters", + "device_scope": "one_selected_device; never sum device peaks", + "custom_sql": "unsupported; select an explicit standalone member", + "work_limit_rows": 100000, + "work_limit_bytes": 67108864, + "budget_scope": "one_query_or_explicit_report_composition" + } + }, + "block": { + "seconds": 0.1289245419902727, + "returned": 3219, + "total": 3219, + "has_more": false, + "truncated": false, + "status": "passed", + "dataset_support": { + "supported": true, + "semantics": "proved_lifecycles_dedup_latest_observation", + "device_scope": "one_selected_device; never sum device peaks", + "custom_sql": "unsupported; select an explicit standalone member", + "work_limit_rows": 100000, + "work_limit_bytes": 67108864, + "budget_scope": "one_query_or_explicit_report_composition" + } + }, + "callstack_analysis": { + "seconds": 0.10561483399942517, + "returned": 60, + "total": 60, + "has_more": false, + "truncated": false, + "status": "passed", + "dataset_support": { + "supported": true, + "semantics": "all_real_stack_bearing_actions_global_identity_weighted_stats", + "device_scope": "one_selected_device; never sum device peaks", + "custom_sql": "unsupported; select an explicit standalone member", + "work_limit_rows": 100000, + "work_limit_bytes": 67108864, + "budget_scope": "one_query_or_explicit_report_composition" + } + }, + "event": { + "seconds": 0.03427075000945479, + "returned": 1, + "total": 1, + "has_more": false, + "truncated": false, + "status": "passed", + "dataset_support": { + "supported": true, + "semantics": "real_events_global_filter_sort_page", + "device_scope": "one_selected_device; never sum device peaks", + "custom_sql": "unsupported; select an explicit standalone member", + "work_limit_rows": 100000, + "work_limit_bytes": 67108864, + "budget_scope": "one_query_or_explicit_report_composition" + } + }, + "freed_block_lifetime": { + "seconds": 0.10654170799534768, + "returned": 2, + "total": 2, + "has_more": false, + "truncated": false, + "status": "passed", + "dataset_support": { + "supported": true, + "semantics": "proved_completed_lifecycles_once", + "device_scope": "one_selected_device; never sum device peaks", + "custom_sql": "unsupported; select an explicit standalone member", + "work_limit_rows": 100000, + "work_limit_bytes": 67108864, + "budget_scope": "one_query_or_explicit_report_composition" + } + }, + "leak_detection": { + "seconds": 0.1085347079206258, + "returned": 325, + "total": 325, + "has_more": false, + "truncated": false, + "status": "passed", + "dataset_support": { + "supported": true, + "semantics": "terminal_dataset_dynamic_survivors_not_confirmed_leaks", + "device_scope": "one_selected_device; never sum device peaks", + "custom_sql": "unsupported; select an explicit standalone member", + "work_limit_rows": 100000, + "work_limit_bytes": 67108864, + "budget_scope": "one_query_or_explicit_report_composition" + } + }, + "memory_peak": { + "seconds": 0.06262266705743968, + "returned": 1, + "total": 1, + "has_more": false, + "truncated": false, + "status": "passed", + "dataset_support": { + "supported": true, + "semantics": "independent_real_counter_peaks_earliest_id", + "device_scope": "one_selected_device; never sum device peaks", + "custom_sql": "unsupported; select an explicit standalone member", + "work_limit_rows": 100000, + "work_limit_bytes": 67108864, + "budget_scope": "one_query_or_explicit_report_composition" + } + }, + "preexisting_live": { + "seconds": 0.03428791603073478, + "returned": 1, + "total": 1, + "has_more": false, + "truncated": false, + "status": "passed", + "dataset_support": { + "supported": true, + "semantics": "owner_event_preexisting_occupancy", + "device_scope": "one_selected_device; never sum device peaks", + "custom_sql": "unsupported; select an explicit standalone member", + "work_limit_rows": 100000, + "work_limit_bytes": 67108864, + "budget_scope": "one_query_or_explicit_report_composition" + } + } + } + }, + { + "mode": "compatibility-v1", + "phase": "cold", + "status": "passed", + "import_wall_seconds": 0.19199895800556988, + "isolated_import_process_peak_rss_raw": 55672832, + "isolated_import_process_peak_rss_kib": 54368.0, + "isolated_import_process_peak_rss_mib": 53.09375, + "rss_scope": "RUSAGE_SELF fresh child high-water through import, before queries; includes interpreter/imports", + "phase_timings": { + "load_representation": { + "seconds": 0.0020353749860078096, + "calls": 1 + }, + "sharded_replay_with_sqlite_and_backfill": { + "seconds": 0.11026433296501637, + "calls": 1 + }, + "compatible_conversion": { + "seconds": 0.020934332977049053, + "calls": 1 + } + }, + "phase_timing_scope": "named wrapped functions, inclusive; sharded replay includes SQLite writes/backfills, NOT parser-only", + "phase_rss": { + "status": "unavailable", + "reason": "No per-phase sampler/isolation; cumulative high-waters cannot be subtracted into load/replay peaks" + }, + "other_phase_timings": { + "status": "unavailable", + "reason": "validation/hash/metadata/publication remainder not separately instrumented" + }, + "reused": false, + "device": 0, + "capacity": 2000, + "total_disk_bytes": 15481668, + "individual_files_bytes": { + "snapshot_expandable.pkl.msinsight/manifest.json": 2884, + "snapshot_expandable.pkl.msinsight/device_0/slice_00004.db": 503808, + "snapshot_expandable.pkl.msinsight/device_0/slice_00000.db": 3096576, + "snapshot_expandable.pkl.msinsight/device_0/slice_00001.db": 4337664, + "snapshot_expandable.pkl.msinsight/device_0/slice_00002.db": 4493312, + "snapshot_expandable.pkl.msinsight/device_0/slice_00003.db": 3047424 + }, + "individual_shard_bytes": { + "snapshot_expandable.pkl.msinsight/device_0/slice_00000.db": 3096576, + "snapshot_expandable.pkl.msinsight/device_0/slice_00001.db": 4337664, + "snapshot_expandable.pkl.msinsight/device_0/slice_00002.db": 4493312, + "snapshot_expandable.pkl.msinsight/device_0/slice_00003.db": 3047424, + "snapshot_expandable.pkl.msinsight/device_0/slice_00004.db": 503808 + }, + "raw_block_rows": 6412, + "unique_block_invariants": 3219, + "repeated_block_rows": 3193, + "repeated_block_cost": { + "repeated_base_integer_cells": 22351, + "physical_disk_attribution": "unavailable: shared SQLite pages/indexes cannot be apportioned to duplicated rows by file-size subtraction" + }, + "errors": [], + "python_version": "3.13.11 | packaged by Anaconda, Inc. | (main, Dec 10 2025, 21:21:08) [Clang 20.1.8 ]", + "platform": "macOS-26.0.1-arm64-arm-64bit-Mach-O", + "exit": 0, + "queries": { + "active_blocks_at_event": { + "seconds": 0.030705500044859946, + "returned": 645, + "total": 645, + "has_more": false, + "truncated": false, + "status": "passed", + "dataset_support": { + "supported": true, + "semantics": "owner_event_active_set_cross_slice_sources", + "device_scope": "one_selected_device; never sum device peaks", + "custom_sql": "unsupported; select an explicit standalone member", + "work_limit_rows": 100000, + "work_limit_bytes": 67108864, + "budget_scope": "one_query_or_explicit_report_composition" + } + }, + "active_memory_callstack_at_event": { + "seconds": 0.02841008303221315, + "returned": 4, + "total": 4, + "has_more": false, + "truncated": false, + "status": "passed", + "dataset_support": { + "supported": true, + "semantics": "owner_event_global_source_groups_before_top_n", + "device_scope": "one_selected_device; never sum device peaks", + "custom_sql": "unsupported; select an explicit standalone member", + "work_limit_rows": 100000, + "work_limit_bytes": 67108864, + "budget_scope": "one_query_or_explicit_report_composition" + } + }, + "allocation": { + "seconds": 0.06630966707598418, + "returned": 8092, + "total": 8092, + "has_more": false, + "truncated": false, + "status": "passed", + "dataset_support": { + "supported": true, + "semantics": "real_events_global_filter_sort_page", + "device_scope": "one_selected_device; never sum device peaks", + "custom_sql": "unsupported; select an explicit standalone member", + "work_limit_rows": 100000, + "work_limit_bytes": 67108864, + "budget_scope": "one_query_or_explicit_report_composition" + } + }, + "allocator_gap": { + "seconds": 0.05707012489438057, + "returned": 1, + "total": 1, + "has_more": false, + "truncated": false, + "status": "passed", + "dataset_support": { + "supported": true, + "semantics": "global_peaks_same_event_counters", + "device_scope": "one_selected_device; never sum device peaks", + "custom_sql": "unsupported; select an explicit standalone member", + "work_limit_rows": 100000, + "work_limit_bytes": 67108864, + "budget_scope": "one_query_or_explicit_report_composition" + } + }, + "block": { + "seconds": 0.11705370806157589, + "returned": 3219, + "total": 3219, + "has_more": false, + "truncated": false, + "status": "passed", + "dataset_support": { + "supported": true, + "semantics": "proved_lifecycles_dedup_latest_observation", + "device_scope": "one_selected_device; never sum device peaks", + "custom_sql": "unsupported; select an explicit standalone member", + "work_limit_rows": 100000, + "work_limit_bytes": 67108864, + "budget_scope": "one_query_or_explicit_report_composition" + } + }, + "callstack_analysis": { + "seconds": 0.10150962497573346, + "returned": 60, + "total": 60, + "has_more": false, + "truncated": false, + "status": "passed", + "dataset_support": { + "supported": true, + "semantics": "all_real_stack_bearing_actions_global_identity_weighted_stats", + "device_scope": "one_selected_device; never sum device peaks", + "custom_sql": "unsupported; select an explicit standalone member", + "work_limit_rows": 100000, + "work_limit_bytes": 67108864, + "budget_scope": "one_query_or_explicit_report_composition" + } + }, + "event": { + "seconds": 0.024453416001051664, + "returned": 1, + "total": 1, + "has_more": false, + "truncated": false, + "status": "passed", + "dataset_support": { + "supported": true, + "semantics": "real_events_global_filter_sort_page", + "device_scope": "one_selected_device; never sum device peaks", + "custom_sql": "unsupported; select an explicit standalone member", + "work_limit_rows": 100000, + "work_limit_bytes": 67108864, + "budget_scope": "one_query_or_explicit_report_composition" + } + }, + "freed_block_lifetime": { + "seconds": 0.09622612502425909, + "returned": 2, + "total": 2, + "has_more": false, + "truncated": false, + "status": "passed", + "dataset_support": { + "supported": true, + "semantics": "proved_completed_lifecycles_once", + "device_scope": "one_selected_device; never sum device peaks", + "custom_sql": "unsupported; select an explicit standalone member", + "work_limit_rows": 100000, + "work_limit_bytes": 67108864, + "budget_scope": "one_query_or_explicit_report_composition" + } + }, + "leak_detection": { + "seconds": 0.09670266695320606, + "returned": 325, + "total": 325, + "has_more": false, + "truncated": false, + "status": "passed", + "dataset_support": { + "supported": true, + "semantics": "terminal_dataset_dynamic_survivors_not_confirmed_leaks", + "device_scope": "one_selected_device; never sum device peaks", + "custom_sql": "unsupported; select an explicit standalone member", + "work_limit_rows": 100000, + "work_limit_bytes": 67108864, + "budget_scope": "one_query_or_explicit_report_composition" + } + }, + "memory_peak": { + "seconds": 0.056008042069152, + "returned": 1, + "total": 1, + "has_more": false, + "truncated": false, + "status": "passed", + "dataset_support": { + "supported": true, + "semantics": "independent_real_counter_peaks_earliest_id", + "device_scope": "one_selected_device; never sum device peaks", + "custom_sql": "unsupported; select an explicit standalone member", + "work_limit_rows": 100000, + "work_limit_bytes": 67108864, + "budget_scope": "one_query_or_explicit_report_composition" + } + }, + "preexisting_live": { + "seconds": 0.024689708021469414, + "returned": 1, + "total": 1, + "has_more": false, + "truncated": false, + "status": "passed", + "dataset_support": { + "supported": true, + "semantics": "owner_event_preexisting_occupancy", + "device_scope": "one_selected_device; never sum device peaks", + "custom_sql": "unsupported; select an explicit standalone member", + "work_limit_rows": 100000, + "work_limit_bytes": 67108864, + "budget_scope": "one_query_or_explicit_report_composition" + } + } + } + }, + { + "mode": "compatibility-v1", + "phase": "reuse", + "status": "passed", + "import_wall_seconds": 0.02520129201002419, + "isolated_import_process_peak_rss_raw": 39157760, + "isolated_import_process_peak_rss_kib": 38240.0, + "isolated_import_process_peak_rss_mib": 37.34375, + "rss_scope": "RUSAGE_SELF fresh child high-water through import, before queries; includes interpreter/imports", + "phase_timings": {}, + "phase_timing_scope": "named wrapped functions, inclusive; sharded replay includes SQLite writes/backfills, NOT parser-only", + "phase_rss": { + "status": "unavailable", + "reason": "No per-phase sampler/isolation; cumulative high-waters cannot be subtracted into load/replay peaks" + }, + "other_phase_timings": { + "status": "unavailable", + "reason": "validation/hash/metadata/publication remainder not separately instrumented" + }, + "reused": true, + "device": 0, + "capacity": 2000, + "total_disk_bytes": 15481668, + "individual_files_bytes": { + "snapshot_expandable.pkl.msinsight/manifest.json": 2884, + "snapshot_expandable.pkl.msinsight/device_0/slice_00004.db": 503808, + "snapshot_expandable.pkl.msinsight/device_0/slice_00000.db": 3096576, + "snapshot_expandable.pkl.msinsight/device_0/slice_00001.db": 4337664, + "snapshot_expandable.pkl.msinsight/device_0/slice_00002.db": 4493312, + "snapshot_expandable.pkl.msinsight/device_0/slice_00003.db": 3047424 + }, + "individual_shard_bytes": { + "snapshot_expandable.pkl.msinsight/device_0/slice_00000.db": 3096576, + "snapshot_expandable.pkl.msinsight/device_0/slice_00001.db": 4337664, + "snapshot_expandable.pkl.msinsight/device_0/slice_00002.db": 4493312, + "snapshot_expandable.pkl.msinsight/device_0/slice_00003.db": 3047424, + "snapshot_expandable.pkl.msinsight/device_0/slice_00004.db": 503808 + }, + "raw_block_rows": 6412, + "unique_block_invariants": 3219, + "repeated_block_rows": 3193, + "repeated_block_cost": { + "repeated_base_integer_cells": 22351, + "physical_disk_attribution": "unavailable: shared SQLite pages/indexes cannot be apportioned to duplicated rows by file-size subtraction" + }, + "errors": [], + "python_version": "3.13.11 | packaged by Anaconda, Inc. | (main, Dec 10 2025, 21:21:08) [Clang 20.1.8 ]", + "platform": "macOS-26.0.1-arm64-arm-64bit-Mach-O", + "exit": 0, + "queries": { + "active_blocks_at_event": { + "seconds": 0.030865582986734807, + "returned": 645, + "total": 645, + "has_more": false, + "truncated": false, + "status": "passed", + "dataset_support": { + "supported": true, + "semantics": "owner_event_active_set_cross_slice_sources", + "device_scope": "one_selected_device; never sum device peaks", + "custom_sql": "unsupported; select an explicit standalone member", + "work_limit_rows": 100000, + "work_limit_bytes": 67108864, + "budget_scope": "one_query_or_explicit_report_composition" + } + }, + "active_memory_callstack_at_event": { + "seconds": 0.028811125084757805, + "returned": 4, + "total": 4, + "has_more": false, + "truncated": false, + "status": "passed", + "dataset_support": { + "supported": true, + "semantics": "owner_event_global_source_groups_before_top_n", + "device_scope": "one_selected_device; never sum device peaks", + "custom_sql": "unsupported; select an explicit standalone member", + "work_limit_rows": 100000, + "work_limit_bytes": 67108864, + "budget_scope": "one_query_or_explicit_report_composition" + } + }, + "allocation": { + "seconds": 0.0677059170557186, + "returned": 8092, + "total": 8092, + "has_more": false, + "truncated": false, + "status": "passed", + "dataset_support": { + "supported": true, + "semantics": "real_events_global_filter_sort_page", + "device_scope": "one_selected_device; never sum device peaks", + "custom_sql": "unsupported; select an explicit standalone member", + "work_limit_rows": 100000, + "work_limit_bytes": 67108864, + "budget_scope": "one_query_or_explicit_report_composition" + } + }, + "allocator_gap": { + "seconds": 0.05702104198280722, + "returned": 1, + "total": 1, + "has_more": false, + "truncated": false, + "status": "passed", + "dataset_support": { + "supported": true, + "semantics": "global_peaks_same_event_counters", + "device_scope": "one_selected_device; never sum device peaks", + "custom_sql": "unsupported; select an explicit standalone member", + "work_limit_rows": 100000, + "work_limit_bytes": 67108864, + "budget_scope": "one_query_or_explicit_report_composition" + } + }, + "block": { + "seconds": 0.11802491697017103, + "returned": 3219, + "total": 3219, + "has_more": false, + "truncated": false, + "status": "passed", + "dataset_support": { + "supported": true, + "semantics": "proved_lifecycles_dedup_latest_observation", + "device_scope": "one_selected_device; never sum device peaks", + "custom_sql": "unsupported; select an explicit standalone member", + "work_limit_rows": 100000, + "work_limit_bytes": 67108864, + "budget_scope": "one_query_or_explicit_report_composition" + } + }, + "callstack_analysis": { + "seconds": 0.10324912506621331, + "returned": 60, + "total": 60, + "has_more": false, + "truncated": false, + "status": "passed", + "dataset_support": { + "supported": true, + "semantics": "all_real_stack_bearing_actions_global_identity_weighted_stats", + "device_scope": "one_selected_device; never sum device peaks", + "custom_sql": "unsupported; select an explicit standalone member", + "work_limit_rows": 100000, + "work_limit_bytes": 67108864, + "budget_scope": "one_query_or_explicit_report_composition" + } + }, + "event": { + "seconds": 0.024870292050763965, + "returned": 1, + "total": 1, + "has_more": false, + "truncated": false, + "status": "passed", + "dataset_support": { + "supported": true, + "semantics": "real_events_global_filter_sort_page", + "device_scope": "one_selected_device; never sum device peaks", + "custom_sql": "unsupported; select an explicit standalone member", + "work_limit_rows": 100000, + "work_limit_bytes": 67108864, + "budget_scope": "one_query_or_explicit_report_composition" + } + }, + "freed_block_lifetime": { + "seconds": 0.09511883289087564, + "returned": 2, + "total": 2, + "has_more": false, + "truncated": false, + "status": "passed", + "dataset_support": { + "supported": true, + "semantics": "proved_completed_lifecycles_once", + "device_scope": "one_selected_device; never sum device peaks", + "custom_sql": "unsupported; select an explicit standalone member", + "work_limit_rows": 100000, + "work_limit_bytes": 67108864, + "budget_scope": "one_query_or_explicit_report_composition" + } + }, + "leak_detection": { + "seconds": 0.09666458296123892, + "returned": 325, + "total": 325, + "has_more": false, + "truncated": false, + "status": "passed", + "dataset_support": { + "supported": true, + "semantics": "terminal_dataset_dynamic_survivors_not_confirmed_leaks", + "device_scope": "one_selected_device; never sum device peaks", + "custom_sql": "unsupported; select an explicit standalone member", + "work_limit_rows": 100000, + "work_limit_bytes": 67108864, + "budget_scope": "one_query_or_explicit_report_composition" + } + }, + "memory_peak": { + "seconds": 0.05659191601444036, + "returned": 1, + "total": 1, + "has_more": false, + "truncated": false, + "status": "passed", + "dataset_support": { + "supported": true, + "semantics": "independent_real_counter_peaks_earliest_id", + "device_scope": "one_selected_device; never sum device peaks", + "custom_sql": "unsupported; select an explicit standalone member", + "work_limit_rows": 100000, + "work_limit_bytes": 67108864, + "budget_scope": "one_query_or_explicit_report_composition" + } + }, + "preexisting_live": { + "seconds": 0.02441979106515646, + "returned": 1, + "total": 1, + "has_more": false, + "truncated": false, + "status": "passed", + "dataset_support": { + "supported": true, + "semantics": "owner_event_preexisting_occupancy", + "device_scope": "one_selected_device; never sum device peaks", + "custom_sql": "unsupported; select an explicit standalone member", + "work_limit_rows": 100000, + "work_limit_bytes": 67108864, + "budget_scope": "one_query_or_explicit_report_composition" + } + } + } + } + ] +} diff --git a/docs/en/database.md b/docs/en/database.md index 14c32cc..234d0bc 100644 --- a/docs/en/database.md +++ b/docs/en/database.md @@ -14,7 +14,11 @@ SnapshotDB is the SQLite database format for persisting PyTorch memory profiling ## Producing a snapshot DB -`pt-snap` analyzes SQLite SnapshotDB files. If you start with a raw PyTorch memory snapshot pickle, import the snapshot with the built-in backend: +`pt-snap` analyzes standalone SQLite SnapshotDB files and explicitly selected +complete validated native/compatible datasets. Closed compatibility-v1 uses +immutable read-only transport; native and standalone policies remain distinct. +[Acceptance, metadata/frame limits and measurements](interop-acceptance.md) +separate schema, source/lifecycle differential and pending GUI evidence. If you start with a raw PyTorch memory snapshot pickle, import the snapshot with the built-in backend: > **Security warning:** Import trusted pickle files only. Pickle deserialization > can execute arbitrary code. The loader rejects non-`builtins` global objects, @@ -386,3 +390,47 @@ CREATE TABLE pt_snap_metadata ( Legacy and externally produced compatible databases may omit this table. They remain queryable, but `pt-snap metadata` reports their metadata as unavailable. + +### Published native datasets + +`import --events-per-slice N` publishes a distinct `pt-snap-native-v2` manifest, +not a compatibility-v1 manifest or another standalone DB layout. Each finalized +shard uses the existing v2 tables plus `pt_snap_block_reference` (version 1), with +`blockId INTEGER PRIMARY KEY`, `stream INTEGER NOT NULL`, `allocCallstack TEXT`, +`freeCallstack TEXT`. These are text lifecycle sources, not structured frames or +an automatic cross-shard JOIN. `pt_snap_metadata` is required and identical in all +members; native manifest identity additionally versions source/device/capacity, +format/manifest/import/extension contracts, independently of CLI release version. + +Real counts use positions per device, preserving original sparse IDs and native +OOM/workspace events. Negative boundaries do not consume capacity. Reader +validation includes full-member hashes and finalized identity/reference consistency +before bounded `event` addressing; it never repairs or migrates files. Aggregate +and lifecycle dataset queries remain unsupported. See the separate +[native manifest protocol](sharded-snapshotdb.md#published-native-dataset-p2) and +[transaction/recovery guide](quickstart.md#optional-import-a-native-sharded-dataset). +P0 compatibility-v1 table order, inline layout and manifest semantics are unchanged. + +### Explicit compatible export + +`import --format msinsight` (alias `compatibility-v1`) publishes physical v1 +`trace_entry_` tables with the ninth column `callstack TEXT`, not views, +with the exact positional block/dictionary schema. It removes staging's unprefixed +`callstack` table and writes `pt_snap_metadata.import_format_version=1`, retaining +isolated `pt_snap_block_reference` text lifecycle sources. Native standalone and +native datasets keep their interning and v2 metadata. Structured ordered frames +are not emitted or inferred from text. + +Inline stack bytes scale with the sum of stack text **over every event**, not +unique stacks; conversion temporarily holds native and inline tables and may leave +SQLite free pages. Capacity does not bound blocks, loading RSS or total disk. +Compatible datasets additionally carry `ptSnap.identity`, metadata, omitted devices +and full member hashes. Recognized whole-artifact metadata is available only after +all attestation checks; external v1 datasets still need no metadata. + +For finalized compatibility datasets ONLY, bare validation and cached analysis +use `mode=ro&immutable=1` after rejecting all live/dangling sidecars and aliases +before the first SQLite open. Closed/checkpointed original WAL bytes/inventory +remain unchanged. This assumes no active writer; native dataset non-WAL policy and +standalone `mode=ro` defaults are unchanged. See +[compatible protocol and no-replace rules](sharded-snapshotdb.md#explicit-msinsight-compatible-export-p2). diff --git a/docs/en/focus-management.md b/docs/en/focus-management.md index 4775e99..ad8f40c 100644 --- a/docs/en/focus-management.md +++ b/docs/en/focus-management.md @@ -7,6 +7,13 @@ select different databases and devices. Use a session override when terminals or agents in the same project must remain isolated. +A target may also be an explicitly selected COMPLETE validated compatibility-v1 +or pt-snap-native-v2 directory or `manifest.json`, not an arbitrary folder/ready +slice. Select the device explicitly; `overview.dataset` supplies format/fingerprint +and real bounds. Valid original no-extension metadata is `unavailable` / +`metadata_missing` (unknown provenance), not a validation bypass. Read-only +skills do not persist focus. See [acceptance and isolated test focus](interop-acceptance.md). + ## Resolution Priority Focus is resolved in this order: @@ -112,6 +119,44 @@ ignore rules automatically. } ``` +## Complete Dataset Focus and Import Compensation + +A complete compatibility-v1 dataset or published `pt-snap-native-v2` directory/ +`manifest.json` can be focused with the same precedence and device rules: + +```bash +pt-snap focus captures/snapshot.pkl.pt-snap-native-v2 --device 0 --json +pt-snap query --template-use event --slice 0 --json +``` + +Native layout is `v2`; compatibility layout is `v1`. All declared members are +validated before writing focus, with no artifact repairs or layout persistence. +Native sparse IDs are addressed as original IDs, not positional counts. Only +bounded `event` queries are supported through dataset focus; aggregate/lifecycle +and cross-slice requests fail explicitly. API session focus is read-only and +`SnapshotAnalyzer` has no import/split facade. + +Import writes project focus in the **current directory**, not the output directory. +`--no-focus` skips it. A native dataset import focus failure, including mutation before an error, +restores the old artifact plus exact prior focus bytes and in-memory config state +(or prior absence) on ordinary compensation. If restoration fails, the error +identifies the retained `.focus.pt-snap-*.recovery` checkpoint and/or dataset +recovery directory. An empty checkpoint with `prior focus existed: False` records +prior absence, not a valid focus JSON. Preserve evidence for manual recovery; +no unconditional I/O/crash atomicity or concurrent-writer lock is promised. +See [native import](quickstart.md#optional-import-a-native-sharded-dataset). + +Explicit compatible import (`--format msinsight`) has the same requested-focus +compensation, but **never replaces** an existing compatible output, even with +force. Only fully attested identical pt-snap caches can be reused; otherwise use +a new output. GUI still requires the same original pickle plus adjacent cache, +although pt-snap focus needs only the complete artifact. All compatible members +are checked for aliases/live/dangling sidecars before any SQLite open; bare +validation and cached queries use `mode=ro&immutable=1` for finalized closed +members, preserving original checkpointed-WAL bytes/inventory. This is not a live +writer lock; native non-WAL checks and standalone defaults remain unchanged. + + ## Error Handling **No focus configured:** diff --git a/docs/en/interop-acceptance.md b/docs/en/interop-acceptance.md new file mode 100644 index 0000000..2725d49 --- /dev/null +++ b/docs/en/interop-acceptance.md @@ -0,0 +1,200 @@ +# Interoperability acceptance and bounded performance + +[中文](../zh/interop-acceptance.md) | English + +## Evidence layers and fixed target + +The ONLY upstream target is **Ascend/msinsight@101f65b877a267ffd5f66ea3834706057ba243e5**. +Protocol/schema validation, isolated source-function checks, synthetic CI, +reading an actual original-producer artifact, and actual GUI/C++ consumption are +DIFFERENT evidence layers. None implies the next. **GUI is human-deferred, +pending/not-run**; the issue stays open. The existing pending note is +[#205](https://github.com/codeasier/pt-snap-cli/issues/205#issuecomment-5987264017). + +The acceptance tools are original interface-facts/reference code, not vendor +copies. Fixed [producer schema](https://api.github.com/repos/Ascend/msinsight/git/blobs/1fe50a5136f85365db55f4632d58f3cf7090431e), +[manifest/backfill](https://api.github.com/repos/Ascend/msinsight/git/blobs/95612509ef990ac6b7034919dae4938cc6acf3ad), +[parser predicates](https://api.github.com/repos/Ascend/msinsight/git/blobs/40046e6a92ff5af776ee55bd4cdd4586745af3c2) +and [salt order](https://api.github.com/repos/Ascend/msinsight/git/blobs/ca421c7b33f3a0050df9e7a01e666f558801bc2f) +are the source of protocol facts. Ascend source is Mulan PSL v2; upstream docs +are separately CC BY 4.0, not automatically covered by this project's MIT lineage. +A port needs snapshot provenance/PR decisions and retained notices. Executable +pickle trust is a SEPARATE fixture review; no new committed pickle is added here. + +## Forward chain: trusted source → physical export → FUTURE GUI + +1. Review/trust the source separately: pickle is executable input, not a sandbox. + Record exact source SHA256, size, selected device and capacity. Default import + stays single-db; native and compatible outputs are explicit distinct modes. +2. Use an ABSENT destination. For the future original-source GUI entrance, the + compatible artifact must be adjacent to the SAME original pickle: + + ```bash + pt-snap import /capture/snapshot.pkl --format msinsight --events-per-slice 2000 --device 0 --no-focus --json + pt-snap overview /capture/snapshot.pkl.msinsight --json + ``` + + `--format compatibility-v1` is the alias. `--output-dir` chooses a different + parent, which does NOT create a direct-directory GUI entrance. Do not overwrite + an existing original/external cache even with force. Only identical recognized + pt-snap ownership/identity can reuse; changed members invalidate attestation. + `cacheHash=SHA256(b"mem_snapshot_parser_v2" + raw_source_bytes)` is distinct from + `ptSnap.identity` (source/options/format/semantic-version/member identity). +3. Compare physical base schemas, dictionaries, ALL nine real-event columns, + real ID/range/action/stream/counter/text values and lifecycle rows against an + existing reviewed original artifact of the SAME source/options. Real block + IDs INCLUDING zero remain exact. Negative block IDs are producer-local tokens: + compare all other six fields INCLUDING state as MULTISETS globally/per shard, + preserving multiplicity; validate each artifact's own token stability. This + does not prove cross-producer negative object identity. Never deduplicate by + address, rewrite IDs, or patch the fixed producer to force equality. +4. **Later, after explicit human authorization**, open the same original pickle + using the exact recorded real GUI build. Complete the checklist below. Physical + equality and salted-hash equality are NOT GUI cache-reuse acceptance. + +## Reverse chain: existing original artifact → complete focus → canonical reference + +No producer is downloaded or executed by the runner. An original artifact must +already exist, be closed/finalized, and have a reviewed receipt. It may have NO +metadata/reference/frame extensions. Its valid metadata result is +`unavailable` / `metadata_missing`, not invented provenance. Invalid metadata, +incomplete validation and unknown reasons stop. + +```bash +pt-snap focus /capture/snapshot.pkl.msinsight/manifest.json --device 0 --json +pt-snap capabilities --json +pt-snap overview /capture/snapshot.pkl.msinsight --json +pt-snap query /capture/snapshot.pkl.msinsight --device 0 --template-use memory_peak --json +pt-snap report peak-memory /capture/snapshot.pkl.msinsight --device 0 --metric reserved --start-id 0 --end-id 100 --limit 20 --json +``` + +`focus` persists a project choice: use an isolated explicitly approved acceptance +project. Diagnostic skills themselves do NOT persist focus, import or install. +For WHOLE-DATASET scope only COMPLETE validated directories/manifests are accepted, +not arbitrary folders or a single ready slice. An explicitly chosen member can +remain a standalone input under standalone policies, NOT whole-dataset scope. Closed compatibility-v1 uses immutable read-only SQLite, +including checkpointed sidecar-free WAL headers; all aliases/live or dangling +WAL/SHM/journal sidecars are rejected before opening. Native retains its stricter +WAL policy; standalone analysis stays `mode=ro` without immutable. + +Optional local runner, from the repository root using the intended interpreter: + +```bash +PYTHONPATH=src PYTHONDONTWRITEBYTECODE=1 python benchmarks/interop_acceptance.py \ + --artifact /capture/snapshot.pkl.msinsight --receipt /evidence/reviewed-receipt.json \ + --source /capture/snapshot.pkl --device 0 --output /evidence/NEW-acceptance +``` + +Optional `--compatible /export/snapshot.pkl.msinsight` compares an EXISTING forward +export; it never generates it. Receipt fields are `revision` (exact full SHA), +`artifactHashes` (ALL relative regular member names → SHA256), and, for the optional +source/forward checks, `fixtureSHA256`, `fixtureBytes`, `saltedCacheHash`. +Receipt authenticity is caller-reviewed evidence, NOT established by hashes. +The source is HASHED ONLY, never unpickled. All member hashes/sidecars are checked +BEFORE SQLite/product opens and after; changed membership fails. The output must +be NEW, canonical, non-symlink and disjoint from inputs. It contains a new physical +canonical-reference DB, report and isolated test project focus, never modifies +external focus/cache and never automatically downloads/executes a producer. + +The small independent oracle materializes bounded rows in memory (32 MiB files, +256 entries, 64 slices, 20000 physical trace rows INCLUDING boundaries, 20000 block +rows, 256 dictionary rows/shard, 1 MiB JSON, optional 2 MiB source). These are +reference-tool limits, NOT a product RSS contract. It compares all real columns, +full completed-free/allocation source identity, canonical lifetimes/latest state, +terminal candidates, global earliest-tie peaks/full and shard ranges, selected +active sets/bytes, all-action stack statistics and global sort/page/exact totals. +The canonical standalone reference is NOT another original producer run. +The console chain verifies identical interpreter/API source across its isolated +CWD, then actual focus/query/report and normalized CLI/API scope/coverage. +Selected points/ranges are recorded, NOT an exhaustive all-event active-set proof. + +## Deterministic CI scenarios and limits + +`pytest tests/test_interop_acceptance.py tests/test_dataset_import_baseline.py + tests/skills tests/test_bundled_skills.py` uses small generated SQLite or existing +reviewed/synthetic runtime fixtures; no network, GUI, NPU, live model or private +absolute baseline paths. Always run `tests/test_fixture_provenance.py` BEFORE any +committed-pickle runtime test. The owning suites below remain part of the gate: + +| Scenario | Deterministic owning regression / interpretation | +| --- | --- | +| Long-lived blocks across many shards; address reuse; pending free; static/preexisting/unknown | `tests/core/test_dataset_attribution.py`, `test_dataset_global.py`; dedup by proved lifetime, not address; free_completed, not free_requested | +| Expandable segments, multi-device, sparse/nonzero native IDs, OOM/workspace | `tests/snapshot/test_sharded_replay.py`, `tests/test_native_dataset.py`, `tests/core/test_msinsight_export.py`; native preserves, v1 explicitly rejects OOM/sparse/nonzero IDs; no upstream OOM coercion or workspace raw-frame-loss claim | +| Ordered frames | Reader-only ptSnapOrderedFrames version 1, actual schema/coverage, finite typed JSON/depth128/order/duplicates; missing/unknown/uncovered → text-only, NO emitter/reconstruction or original-produced-frame claim | +| Cache/source/options/format/semantic identity; member mutation; no-replace external cache | Existing native/compatible import suites plus acceptance tests; GUI-derived DB mutation invalidates attestation, NOT proof of producer rerun | +| Write/backfill/conversion/source recheck/publication/focus/rollback failures | `tests/core/test_dataset_import_failures.py`, `test_import_backend_failures.py`, `test_msinsight_export.py`, sharded replay suite; assert ACTUAL old destination/focus/recovery bytes, not just exceptions | +| All eleven built-ins, CLI/API/report, paging and budgets | New physical reference + `test_dataset_global.py`; custom SQL/built-in-name overrides unsupported; borrowed LRU/terminal closure/handler cleanup unchanged | + +Use the SAME YAML catalog's `dataset_support` and optional declared row fields. +`scope.range_complete` (ranges) and `scope.source_coverage.range_complete` (points), +allocation/frame-source coverage and result `has_more`/`truncated` are independent. +Dataset canonical `source_stack_id` differs from local stack ID provenance; +representatives prefer nonempty text. Missing references, captured literal labels +and non-NULL empty activity groups are not interchangeable. `callstack_analysis` +semantics v2 legacy `alloc_count` counts ALL real stack-bearing actions; +`total_size` is activity, not live bytes. Percentages retain included group bytes (standalone SQL or dataset post-merge) +after filters/top-N BEFORE caller row caps (#179), not the whole active counter. +Reports use gap/attribution/active coverage at the SAME selected metric event. +One query/report shares diminishing time and cumulative 100000 fetched/output rows +/64 MiB serialized values, NOT RSS. Failure yields no partial false global result. + +## Performance: same reviewed fixture, three modes + +The bounded measured fixture is `snapshot_expandable.pkl`, 581140 bytes, +SHA256 `3afc9d1c5ef4ca4b417e58c0830e9eb8a913eb9459f4088b8c66c22325c68c40`, +device0/capacity2000. `baseline_import.py --samples 8k` selects a DIFFERENT +multi-device fixture, so it is not this original-reference comparison. + +```bash +PYTHONPATH=src PYTHONDONTWRITEBYTECODE=1 python -m pytest tests/test_fixture_provenance.py +PYTHONPATH=src PYTHONDONTWRITEBYTECODE=1 python benchmarks/dataset_import_baseline.py \ + --source tests/fixtures/snapshots/snapshot_expandable.pkl \ + --sha256 3afc9d1c5ef4ca4b417e58c0830e9eb8a913eb9459f4088b8c66c22325c68c40 \ + --device 0 --capacity 2000 --output /evidence/NEW-three-mode --trusted-pickle +``` + +This is explicit trusted-pickle execution consent, not an automatic diagnostic +step. No installs/repointing, large default sample, original-producer rerun or +cleanup. Each cold/reuse measurement uses a fresh isolated child; cold means NEW +output/no warmup, NOT cleared OS page cache. Reuse requires this run's unchanged +owned cold result; cache miss cannot rebuild/overwrite. Same-source/mode/phase +ownership is checked BEFORE pickle execution. Exact commands/exits/error logs, +source/head/script hashes, Python/platform and all eleven supported query latencies +are recorded. [Machine-readable local observation](../dataset-performance-baseline.json) +contains sanitized measured data, not speed assertions or CI numerical oracles. +Supported-query scope is not identical across modes: this observation's default +standalone `allocation` returns 8094 rows (8092 real + 2 synthetic boundary rows), +while dataset-global `allocation` returns only 8092 real rows. Event lookup selects +an explicit real ID. Preserve these differences when interpreting latency/row +counts; availability of the same template name does not prove equal logical scope. + +Whole import wall and actual inclusive wrapped load/replay/SQLite/backfill/conversion +function times are distinct. `RUSAGE_SELF` fresh-child high-water is captured through +import BEFORE queries; Darwin raw bytes are divided by1024 to KiB, then1024 to MiB. +It includes interpreter/import overhead. Per-phase load/replay RSS and uninstrumented +remainder timings are **unavailable**, with reasons; do not subtract cumulative +high-waters or use parent/previous-child `RUSAGE_CHILDREN`. Total disk, individual +shards, repeated block rows and query latency are measured; per-row physical page +cost is unavailable. Pickle still loads fully; shards repeat long-lived blocks and +compatible inline text. Query hashing/scans may dominate. No unmeasured speed/RSS/ +disk-saving promise. The existing original run's single cold wall0.160545s has NO +RSS/phase measurements and is not a controlled comparative performance baseline. + +## FUTURE GUI checklist — every row PENDING / NOT-RUN + +- [ ] Exact real version/build/commit/package SHA, interpreter/server provenance, + environment and exact startup/open command; don't invent executable syntax. +- [ ] SAME original pickle path/hash/size and adjacent complete artifact; all + manifest/member/checksum inventories BEFORE and AFTER opening, source unchanged. +- [ ] UP_TO_DATE/cache-reuse logs AND absence of producer/parser script invocation, + with process/log observation that could detect rebuilding; hash alone is not proof. +- [ ] Curves/counters/ranges/real IDs excluding boundaries at recorded points; + blocks/sizes/states and cross-shard allocation/free_completed source/details. +- [ ] Ordered-frame claims only where actual recognized coverage exists; original + text-only artifacts remain text-only. Record unavailable views honestly. +- [ ] Inspect derived allocation-cache mutation separately: it may change DB bytes + without producer rerun and invalidate ptSnap ownership attestation. Preserve both + inventories, never repair/overwrite an original cache just to make reuse pass. + +Synthetic/schema/original-artifact checks and static skill grades cannot tick +these rows. No GUI/C++ execution/compile occurred in this acceptance package. diff --git a/docs/en/querying.md b/docs/en/querying.md index 074496f..60d92ed 100644 --- a/docs/en/querying.md +++ b/docs/en/querying.md @@ -4,6 +4,14 @@ Run memory analysis queries against your snapshot database. +Use the catalog's SAME YAML `dataset_support`/declared row fields and an explicit +validated target/device/scope. [Offline acceptance and measured query latency](interop-acceptance.md) +separate source/range/frame coverage from `has_more`/`truncated`, explain text-only +downgrades and all-action stack statistics, and retain included-byte denominators. +Bounded `report peak-memory --start-id ... --end-id ...` uses the SAME selected +metric event for gap/active attribution/coverage; custom SQL/overrides remain +unsupported for whole datasets. Measurements are not speed assertions. + ## The Query Command ```bash @@ -16,7 +24,7 @@ pt-snap query [DB_PATH] [--template-use ] [--params ] \ | Flag | Description | |------|-------------| -| `db_path` | SQLite database file path (optional if focus is configured) | +| `db_path` | Standalone SQLite file or COMPLETE validated dataset directory/manifest (optional if focus is configured) | | `--template-use` | Query template name (required unless using `--list` or `--template-info`) | | `--params` | Query parameters in JSON format | | `--device` | Device ID | @@ -24,7 +32,7 @@ pt-snap query [DB_PATH] [--template-use ] [--params ] \ | `--category` | Filter templates by category: `basic`, `statistical`, `business` | | `--template-info` | Show template details (parameters, output schema, and field semantics) | | `-n` | Maximum displayed rows; zero or a negative value means unlimited. Separate from `--timeout`. | -| `--exact-total` | Run a `COUNT` of the matching set. Default `total` is the returned-row count. | +| `--exact-total` | Count the complete matching set: standalone SQL COUNT or dataset post-merge count. Default `total` is returned rows. | | `--timeout` | Wall-clock budget for one `query` / `QueryService` call (page query and optional `--exact-total` COUNT share it) via a SQLite progress handler. `<= 0` disables. Default: `PT_SNAP_QUERY_TIMEOUT` or unlimited. The environment variable applies to every template query, including `report peak-memory`. | | `--json` | Emit machine-readable JSON (execute, `--list`, and `--template-info`) | @@ -78,6 +86,161 @@ group identity. Missing/static/preexisting groups have no captured allocation stack to retrieve; omit `stack_bytes` on the original query to see their full display labels. Results are read-only and summaries do not change the database. +## Dataset-global built-in support (P3) + +Complete native-v2 and compatibility-v1 datasets share the following **one-device** +query support matrix, also exposed as each catalog/template-info `dataset_support`. +Device selection stays explicit → focused → first discovered; peaks are never summed +across devices and there is no shared cross-device event order. + +| Template | Dataset semantics before the final output window | +| --- | --- | +| `event`, `allocation` | Filter all selected real events, globally sort with the declared ID tie-break, then offset/limit/max_rows; negative boundaries excluded | +| `memory_peak` | Independent allocated/active/reserved maxima over real events; earliest real ID wins ties; `start_id/end_id` may cross shards | +| `allocator_gap` | The same independently selected global events, with each gap computed from counters **at that event** | +| `active_blocks_at_event`, `active_memory_callstack_at_event` | Route a real event to its owner and resolve full cross-shard sources before grouping/top_n (details below) | +| `preexisting_live` | Owner-event preexisting occupancy, not a leak count | +| `block` | Merge proved allocation-event lifetimes and stable negative tokens once, then apply filters/order/page; latest containing-shard state is an observation, not state at arbitrary E | +| `freed_block_lifetime` | Deduplicated captured allocations with a proved `free_completed` source; event-ID distance buckets, not time | +| `leak_detection` | Captured dynamic lifetimes still present without completion in the **terminal dataset shard**, not union of per-shard unfreed lists; survival is only a candidate, not a confirmed leak; `--slice` is rejected | +| `callstack_analysis` | All real stack-bearing trace actions, including free/segment/workspace, merged by full canonical source identity before global thresholds/ranking; weighted `avg_size=total_size/alloc_count` | +| Runtime overrides/custom templates/arbitrary SQL | Explicitly unsupported on datasets, even when named after a built-in; pass an explicit standalone member to use local SQL | + +`memory_peak` and `allocator_gap` semantics version **2** exclude negative synthetic +rows in **standalone** SQL too. `callstack_analysis` version **2** retains the legacy +`alloc_count` column name but does **not** mean allocation count: v1 counts all real +non-NULL-text rows; v2 counts all real non-NULL stack references, retaining absent/NULL +joined text as one distinct missing group. Empty text counts as its own group, separate +from missing and captured literal labels. Both standalone layouts now group full text, +not local v2 IDs. Datasets also accept explicitly covered ordered +raw arrays as stack evidence; missing text does not change raw-array identity. Size +sums include nonallocation actions and are not instantaneous live occupancy. + +The shared YAML output schemas declare every optional dataset-only row column; +standalone row shapes stay unchanged. `block` and dataset `leak_detection` expose +`lifecycle_id`, proof/source statuses, source payloads, latest observation slice/state +scope, category and terminal survival; leak rows additionally retain `requestedSize`, +`state` and `freeEventId`. These observations and proofs are not leak confirmation. +Dataset statistics expose `source_stack_id`, `stack_kind`, `text_kind`, +`stack_event_id` and `frames_status`. The representative real event prefers nonempty +captured text when available but is not the canonical group identity and need not be +an allocation action; otherwise it may have empty/NULL text. `text_kind` is provenance +only and never changes raw-array identity. Retrieve actual text/arrays using `event` +on that same dataset/device; `frames_status` is not frame reconstruction. + +```bash +pt-snap query '' --device 0 --template-use memory_peak --params '{"start_id":100,"end_id":700}' --json +pt-snap query '' --device 0 --template-use event --params '{"min_id":100,"max_id":700,"limit":20,"offset":20}' --exact-total --json +pt-snap query '' --device 0 --template-use leak_detection -n 20 --exact-total --json +pt-snap report peak-memory '' --device 0 --metric reserved --start-id 100 --end-id 700 --timeout 10 --json +``` + +Use actual bounds. Ranges may include sparse gaps; they never create an event. +An empty real range has NULL peak values/IDs. Outside/negative/reversed range +endpoints fail; `event(id=gap)` may return no rows, while active-at-gap fails. +Global filters/dedup/source identity/thresholds/sort precede the final window. +`exact_total` ignores row caps; default total is returned rows with honest exactness +flags. Result `has_more/truncated` describes the window, not source/range completeness. +`scope` records requested bounds, actual real-event coverage where applicable, +involved `slice_indices`, device, fingerprint and boundary exclusion. Source coverage +is separate. Unproved historical lifecycle identities are shard-local, explicitly +marked, never guessed by address/local stack ID and never asserted as proved leaks. + +Dataset reads detach batches before foreign-owner lookups, using the borrowed bounded +Context LRU. One diminishing deadline includes all shards, sources, grouping and exact +totals; reports explicitly share **one** budget across their three queries. Cumulative +fetched/output work is limited to **100000 rows / 64 MiB serialized values**, including +repeated source reads. Exceeding either fails the entire operation, never an alleged +complete partial result. Materialization/sort are bounded by this work ceiling, **not** +by max_rows. This is not a process RSS limit: Python/SQLite, one batch and individual +cells have overhead. Manifest hashing/validation remains proportional to artifact size +and phase-checked, not hard-preempted. No temporary merge DB, repair or artifact write. + +## Dataset point-event attribution + +A complete compatibility-v1 or native-v2 directory/manifest supports `event` +addressing plus `active_blocks_at_event` and `active_memory_callstack_at_event` +with a real `event_id`. The latter two route one device's actual event and use +its shard's complete lifecycle observations **after** that forward event: +`allocEventId <= E` and completed `freeEventId > E` (or `-1`). `free_requested` +does not remove active memory. Negative boundary rows represent the instant +before a shard's first event and are not query endpoints. Native sparse IDs are +not renumbered: `event(id=gap)` may return no rows inside an addressed slice; +active attribution at a nonexistent ID fails instead of inventing event 0. + +```bash +pt-snap query '' --device 0 --template-use active_memory_callstack_at_event --params '{"event_id":700,"top_n":20}' --json +pt-snap query '' --device 0 --template-use active_blocks_at_event --params '{"event_id":700,"limit":10}' --exact-total --json +``` + +Replace the path/device/event with actual values. Foreign alloc/free events and +full v1 inline or v2 local-ID stacks are resolved **before** grouping/top_n, in +batches of at most 256 distinct event IDs per owning shard. No foreign real rows +are inserted. Original artifacts need neither metadata nor reference tables. +Lifetimes use dataset generation/device/alloc-event or stable negative identity, +not addresses; `state_scope=slice_observation` warns that stored `state` is not +necessarily the pending-free state at E. Block rows expose `allocation_source`, +`free_source`, their statuses and `lifecycle_id`. Missing dynamic sources retain +bytes and dynamic category, separate from static/preexisting unknown history. +`free=-1` means live **or unknown**, not a proven leak. + +Dataset groups always expose full `source_stack_id`/`stack_id`, `stack_kind` and +`stack_event_id`. Default no-extension `event` rows keep their original nine +values; source/frame row fields appear with `stack_bytes>=0` or a recognized +ordered-frame declaration. `scope.source_coverage` is still supplied. Both dataset +layouts use full-text canonical identity; v2 looks up the owning slice's actual +local ID, returned as source `local_stack_id`/`slice_index` provenance, not a +cross-shard grouping key. Covered ordered arrays use only exact raw-frame identity, +independent of formatted text availability; `text_kind` reports that availability +separately. Explicitly covered arrays (including empty arrays) are captured raw +evidence. Groups prefer an available captured-text representative for retrieval. Nonempty whitespace and a captured literal `[missing callstack]` +are not missing. Never merge by local IDs or shortened display labels. `stack_bytes` +shortens display only; use `event(id=stack_event_id)` on the same dataset/device +for full text. Ordered raw frames, when explicitly covered, also participate in +identity; no text parsing reconstructs frames. + +`scope.source_coverage` is independent of `has_more`/`truncated`/exact row totals: +active-query coverage describes **all filtered blocks before top_n/paging**, with +active/dynamic bytes, resolved allocation/free counts, captured dynamic bytes, +unknown/preexisting counts, ordered-frame coverage and degradation. For `event`, +its coverage scope is only `returned_events`. A complete manifest/range does not +prove complete historical sources or frames. Percentages still use included +bytes **after dynamic top_n but before max_rows**, not the dataset active counter. +Exact totals ignore row/rank caps, but do not change that percentage denominator. + +One call shares its deadline across source batches, grouping and exact totals. +Manifest validation/hash and Context setup elapsed time are charged, but synchronous +filesystem I/O/validation is checked at phase boundaries, not hard-preempted; no +OS-level time guarantee is claimed. SQL uses cleared-on-exit progress handlers. +The Context LRU remains bounded (default four) and borrowed caches remain caller-owned. +No merged temporary database is created. `ReportService.event_attribution(E, ...)` +reuses this same query path for a supplied event. Dataset-global queries and reports +follow the explicit support matrix above; arbitrary SQL/custom overrides never inherit +those merge semantics merely by using a built-in name. + +### Optional ordered-frame reader contract + +This reader-only pt-snap extension is new, not an existing original-producer +feature. Current native/compatible writers emit **no** ordered-frame tables. +Declare `extensions.ptSnapOrderedFrames={"version":1}` in the manifest and use +isolated actual tables in each covered source shard: + +- `pt_snap_frame_coverage(eventId INTEGER PRIMARY KEY, frameCount INTEGER)` +- `pt_snap_frame(eventId INTEGER, frameIndex INTEGER, frameJson TEXT, PRIMARY KEY(eventId,frameIndex))` + +Raw objects must be finite standard JSON, with nesting depth at most 128; deeper +or nonfinite/unparseable input degrades safely, including recursion failures. +Each covered real event needs nonnegative integer `frameCount`, exactly indices +`0..frameCount-1`, and JSON **objects** preserving all raw typed fields. Order and +repeated frames are retained; empty arrays require an explicit count zero row. +Only recognized version/schema and valid per-event coverage produce `frames` +with `frames_status=ordered`. Unknown version, missing schema/rows, invalid JSON +or incomplete coverage degrade to `frames=null`, `frames_status=text_only`; +formatted text is never split/reversed into raw frames. P0/overview's dataset-wide +`structured_frames=false` remains a conservative validation capability, not a +claim about the scoped source payload. Tree rendering and GUI acceptance are not +provided by this extension interface. + ## Query Templates Templates are organized into three categories. Use `pt-snap capabilities --json` for the full catalog (CLI version, every template contract, and bundled skills), or `pt-snap query --list` to see names and descriptions. Filter with `--category`. @@ -273,7 +436,7 @@ This is useful because `reserved` may peak at a different event from `active` or For a higher-level summary, use the report command: ```bash -pt-snap report peak-memory [db_path] [--device ] [--metric active|allocated|reserved] [--include-static|--exclude-static] [--limit ] [--json] +pt-snap report peak-memory [db_path] [--device ] [--metric active|allocated|reserved] [--include-static|--exclude-static] [--limit ] [--start-id ] [--end-id ] [--timeout ] [--json] ``` Examples: @@ -295,7 +458,12 @@ The report command combines: - `allocator_gap` - `active_memory_callstack_at_event` -and prints either a human-readable summary or JSON. +and prints either a human-readable summary or JSON. `--start-id/--end-id` restrict +peak selection; attribution still resolves the complete active set at that selected +event, including earlier allocations. `--timeout` (or `PT_SNAP_QUERY_TIMEOUT`) is +one diminishing **report-wide** deadline, not three reset per-query timeouts. +JSON adds peak `scope`, separate attribution `source_coverage`, `timeout_s`, and +`budget_scope="report_composition"`. The existing `callstack_groups` and `percent_of_active_blocks` values are preserved. JSON also exposes attribution `has_more`, `truncated`, `total_is_exact`, and @@ -332,8 +500,10 @@ An empty trace returns no groups, zero included bytes, and unknown coverage. `-n` still caps `rows` and `returned`. By default `total` equals `returned` (this page's row count) and `total_is_exact` is true only when the page is the complete matching set (`has_more` is false and `offset` is 0). -`--exact-total` runs a `COUNT` of the matching set (ignoring `limit` / -`offset` / `top_n`) and sets `total_is_exact` to true. When a finite +`--exact-total` counts the matching set (ignoring `limit` / `offset` / `top_n`) +and sets `total_is_exact` to true. Standalone queries use SQL `COUNT`; dataset +built-ins count the fully merged/deduplicated matching set before its global window, +not capped shard-local rows. On the standalone path, when a finite trailing `LIMIT` is in effect, the executor fetches one extra row to set `has_more` without that count. A finite inner `top_n` (for example inside a CTE) is not a trailing `LIMIT`: when that ranked window is full, @@ -349,7 +519,9 @@ change the row cap. A non-numeric `PT_SNAP_QUERY_TIMEOUT` is `INVALID_PARAMETER`, not `QUERY_FAILED`. `effective_params` is the validated parameter set after defaults and `choices` -normalization. When `-n` is set, `limit` is the trailing SQL LIMIT after the +normalization. Dataset `limit` reports the final global output-window cap (with the +same min-of-positive-caps rule), not a per-shard SQL LIMIT; global filters and merges +are already complete. On the standalone path, when `-n` is set, `limit` is the trailing SQL LIMIT after the same merge the executor applies: `min` of a declared template `limit` and `-n`, or `-n` appended when the rendered SQL has no trailing LIMIT. Inner caps such as `top_n` stay their own parameters; they do not rewrite diff --git a/docs/en/quickstart.md b/docs/en/quickstart.md index ba08d6b..d928522 100644 --- a/docs/en/quickstart.md +++ b/docs/en/quickstart.md @@ -4,6 +4,12 @@ Get up and running with `pt-snap-cli` in a few minutes. +For explicit native/compatible import acceptance, the SAME reviewed-fixture +three-mode baseline and the pending GUI checklist, see +[interoperability acceptance](interop-acceptance.md). Default import remains one +standalone DB; `--events-per-slice` opts into native, `--format msinsight` into +compatibility-v1. Trusted import consent is separate from read-only diagnosis. + ## Installation ```bash @@ -49,6 +55,100 @@ copying raw events directly. The resulting SnapshotDB records event-by-event the supported input for `pt-snap query` and `pt-snap report`. This replay behavior is part of the command workflow; snapshot runtime Python modules are not a public API. +### Optional: Import a Native Sharded Dataset + +```bash +pt-snap import snapshot.pkl --events-per-slice 50000 --output-dir captures --json +pt-snap overview captures/snapshot.pkl.pt-snap-native-v2 --json +pt-snap query --template-use event --slice 0 --json +``` + +Without `--events-per-slice`, import still produces the existing single DB. With +it, the output is `/.pt-snap-native-v2/`, or the +same directory beside the source if `--output-dir` is omitted. `--format +pt-snap-native-v2` is optional in this mode; `--format single-db` forbids a capacity. +Native output is **not** an msinsight-compatible export. Explicit `--format +msinsight` (alias `compatibility-v1`) selects the separate compatible mode below. + +Capacity counts real chronological positions **per device**, not `max(id)+1` or +negative synthetic boundaries. Original sparse/nonzero IDs, OOM and workspace +events remain native. `--device` selects one event-bearing device; otherwise all +such devices are imported. Empty/static-only and unselected device positions are +reported in `omitted_devices`; selecting an empty device or an all-empty input +fails. Sharding does not bound pickle loading, registry/block rows or peak RSS. + +Reuse requires a validated complete manifest and **every** closed member, with +matching source SHA256, device selection, capacity, native format, manifest, +semantic import, extension and metadata/layout contracts. CLI release version +changes alone do not rebuild. Native identity is separate from future compatible +format identities and from msinsight's salted `cacheHash`. JSON adds `dataset_path`, +`devices`, `slice_count`, `format`, `omitted_devices`, `reused` and `rebuilt`; +`db_path` denotes the dataset directory in this mode. + +A matching target is reused. A different/corrupt recognized native artifact is +preserved unless `--force` explicitly requests replacement. Unknown directories, +extra unrelated files, malformed manifests and symlink targets are never adopted +or deleted, even with force: choose a new `--output-dir`. All devices/shards are +staged beside the target, finalized, closed, metadata-attested and validated; the +source is hashed again before publication. No partial shard is a reusable cache. + +New publication uses an exclusive directory rename. Force replacement first +renames the old artifact into an owned recovery directory, then publishes the +new one; **there is a transient path gap, not a single atomic swap**. Requested +project focus is part of compensation, including a focus write that mutates +before raising: ordinary failure restores old artifact and exact old focus +bytes/state (or previous absence). `--no-focus` skips focus entirely. If rollback +itself fails, recovery paths are preserved and explicitly reported. Do not delete +them before inspecting the evidence. Arbitrary I/O failures, concurrent writers +and crashes do not have unconditional atomic restoration guarantees. + +Dataset focus supports bounded `event` addressing and point-event active block/ +callstack attribution with shared cross-shard sources. Dataset-global peak/list/leak +aggregates remain explicit errors, not one-shard approximations. See +[point-event coverage](querying.md#dataset-point-event-attribution) and the +[native manifest contract](sharded-snapshotdb.md#published-native-dataset-p2). +`SnapshotAnalyzer` remains analysis-only, without import/split methods. + +### Optional: Export an msinsight-Compatible Dataset + +```bash +pt-snap import snapshot.pkl --format msinsight --json +pt-snap overview snapshot.pkl.msinsight --json +pt-snap query --template-use event --slice 0 --json +``` + +This **explicit** mode targets `Ascend/msinsight@101f65b877a267ffd5f66ea3834706057ba243e5` +and its original-pickle import entrance, not direct directory import. Keep the +same original pickle beside `.msinsight/` for GUI discovery; +pt-snap itself can analyze a relocated complete artifact without pickle. Optional +`--events-per-slice N` defaults to `500000`; `--output-dir` selects another parent, +not an independently supported GUI entrance. `--device` selects an event-bearing +device and omissions are reported. Nonzero/sparse real IDs, OOM/unknown actions, +empty/static-only selections fail, without silent remapping. Native mode still +preserves sparse IDs/OOM/workspace. Trusted pickle and full-input memory costs apply. + +Base tables are physical inline-v1 tables; metadata has import format `1`. +Callstack text is copied once **per event**, not once per distinct stack, so disk +and temporary conversion space can grow substantially. Native interning remains +unchanged. No ordered structured-frame capability is invented from formatted text. +Bounded `event` and point-event active block/callstack dataset queries are +available; global aggregate/report selection remains unsupported. + +Compatible publication is **no-replace**: external/original/user caches are never +overwritten or deleted, even under `--force`. A complete, attested, identical +pt-snap cache is reused (also with force); otherwise choose a new `--output-dir`. +Source content, sourceFile, options, all member hashes and contract versions must +match independently of salted `cacheHash`. A GUI-derived cache modification can +invalidate this attestation, without implying producer rerun. All staged members +close/finalize/validate before exclusive publication and requested focus; ordinary +failure compensates exact old focus bytes, with recovery evidence if rollback fails. +Standalone/native force policies remain unchanged. `SnapshotAnalyzer` is analysis-only. + +**GUI acceptance is pending/not run.** Complete/ready, safe paths/device tables, +and derived allocation-cache availability/buildability also matter; matching hash +alone proves neither GUI reuse nor display parity. Details and fixed source evidence: +[compatible protocol](sharded-snapshotdb.md#explicit-msinsight-compatible-export-p2). + ### Optional: Split a Snapshot Use `pt-snap split` when you need smaller, independently replayable files. Split diff --git a/docs/en/sharded-snapshotdb.md b/docs/en/sharded-snapshotdb.md new file mode 100644 index 0000000..3948d19 --- /dev/null +++ b/docs/en/sharded-snapshotdb.md @@ -0,0 +1,460 @@ +# Sharded SnapshotDB protocols (compatibility v1 and native v2) + +[中文](../zh/sharded-snapshotdb.md) | English + +## Scope and versions + +This defines artifact contracts, shared validators and an **explicit** compatible +CLI exporter below. The [P3 built-in query matrix](querying.md#dataset-global-built-in-support-p3) +defines supported cross-slice execution separately. Existing `import` defaults to one +native v2 DB; `split` still produces replayable pickle/JSON. Existing single-DB +v1/v2 analysis retains its entry points; P3 real-event peak/gap and all-action full-text +stack-statistics semantics are explicitly versioned in the querying guide. The P1 reader below additionally accepts a +complete compatibility-v1 manifest or directory as focus. The `complete` gate +applies only to datasets, not to legacy single-DB analysis. + +`pt_snap_cli.core.dataset_contract` provides `parse_manifest`, +`validate_event_ids`, `validate_dataset`, immutable manifest/device/slice models, +`QueryScope`, and `DatasetContractError` (`location`, `code`). Producers can share +the same prepublication contract with readers. Validation never loads pickle, +creates missing databases, migrates SQLite, or writes focus. + +Separate version axes: + +- `schemaVersion=1`: manifest and compatibility base-table contract defined here. +- `pt_snap_metadata.import_format_version`: existing pt-snap importer format, + `2` for native and `1` for compatible inline export; **not** the manifest version + and not required for external artifacts. +- Future structured-frame extensions have their own version. Unknown extensions + do not invalidate base reading or imply structured-frame support. + +## Layout and manifest + +The producer convention is `.msinsight/manifest.json` +with `device_/slice_.db` relative to that artifact directory. +An explicitly supplied relocated directory can be validated; `sourceFile` records +origin, not a path to open. Never load the source pickle during inspection. + +```json +{ + "schemaVersion": 1, + "status": "complete", + "sourceFile": "/capture/snapshot.pkl", + "cacheHash": "producer-specific-opaque-identity", + "eventsPerSlice": 2, + "devices": { + "0": { + "eventCount": 4, + "sliceCount": 2, + "readySlices": [0, 1], + "slices": [ + {"index": 0, "startEventId": 0, "endEventId": 1, "file": "device_0/slice_00000.db", "ready": true}, + {"index": 1, "startEventId": 2, "endEventId": 3, "file": "device_0/slice_00001.db", "ready": true} + ] + } + } +} +``` + +All fields shown are required. Integers are not booleans/floats; counts/ranges +fit signed 64-bit integers, version/index/sliceCount/device IDs fit nonnegative +signed 32-bit integers. Device keys are canonical decimal IDs (`0`, not `00`). +`sourceFile` is nonempty. `cacheHash` is an opaque producer token, possibly empty +for an external artifact: it is **not** assumed to be SHA-256 or proof of source +content. A future cache must distinguish source content and producer/format, +base/extension versions, selected devices, capacity, and finalized generation; +source path, size/mtime, `cacheHash` alone, or a per-slice index is insufficient. +This P0 module performs no cache reuse. + +`building` permits partial readiness for `parse_manifest(..., +require_complete=False)` inspection only. It still declares the complete planned +range layout. `complete` requires all slices ready **after identity/lifecycle +backfills finish**. Readiness indices must be unique, in range, and agree with each +slice's boolean. `validate_dataset` accepts only complete datasets with every +file present. Missing files are not created. Duplicate JSON members are rejected. + +Per device, slices are in index order starting at 0; inclusive ranges continuously +cover `0..eventCount-1` without overlap/gaps. Each range has at most +`eventsPerSlice` real events (short intermediate slices are legal). Count real +positions, not `max(id)+1`. Compatibility v1 **rejects nonzero-start, sparse, +reordered or duplicate original real IDs**; no silent renumbering. Empty devices +and static-only devices cannot be represented: reject a requested such device, +never invent event 0. An all-empty dataset is rejected. Other devices may be +explicitly selected, but omission must be disclosed by the producer. + +Paths must be exactly the declared device/index relative path: no absolute/drive/ +UNC paths, `..`, `.`, empty components, backslashes, NUL, device mismatch, aliases +or symlink members/parents. Validation uses read-only SQLite (`mode=ro`) and closes +all handles, including failures. These checks are not an OS sandbox or a +concurrent-publication lock. + +## Fixed base SQLite schema + +Each slice has **actual tables**, not compatibility views, for exactly its device: +`trace_entry_`, `block_`, and `dictionary`. Their column order is normative +because the upstream reader uses positional `SELECT *`. Native v2 `callstackId` +is not a compatible substitute for inline text. + +| Table | Columns, in order (SQLite declared type) | +| --- | --- | +| `trace_entry_` | `id INTEGER PRIMARY KEY`, `action INTEGER`, `address INTEGER`, `size INTEGER`, `stream INTEGER`, `allocated INTEGER`, `active INTEGER`, `reserved INTEGER`, `callstack TEXT` | +| `block_` | `id INTEGER PRIMARY KEY`, `address INTEGER`, `size INTEGER`, `requestedSize INTEGER`, `state INTEGER`, `allocEventId INTEGER`, `freeEventId INTEGER` | +| `dictionary` | `table TEXT`, `column TEXT`, `key TEXT`, `value TEXT` | + +The dictionary stores decimal integer keys as text, with the actual suffixed +table name and column name. Each slice includes all these exact mappings: + +| Column | Key → value | +| --- | --- | +| `trace_entry_.action` | `0 → segment_map`, `1 → segment_unmap`, `2 → segment_alloc`, `3 → segment_free`, `4 → alloc`, `5 → free_requested`, `6 → free_completed`, `7 → workspace_snapshot` | +| `block_.state` | `-1 → inactive`, `0 → active_pending_free`, `1 → active_allocated` | + +No unknown base action/state is accepted. In particular, native pt-snap's `oom=8` +is outside this pinned compatibility baseline; a future exporter must reject it +or define a separately negotiated extension, not relabel it. Metrics and sizes +are bytes; event IDs are chronological identities, **not timestamps**. + +## Boundary events and block lifetimes + +- Real events have nonnegative original IDs. `eventCount`, capacity, ranges and + real-event peaks count/filter only `id >= 0`. +- Negative trace IDs are separately identified synthetic left-boundary state rows + (`segment_map=0` or `segment_alloc=2`). They may have large memory totals but + must not compete for a real-event peak or consume capacity. IDs are slice-local; + `(device, slice, negative ID)` identifies a boundary row. Repeated boundary + segments do not represent new real allocations. Validation reports + `real_event_count` and `boundary_event_count` separately; it does not execute peaks. +- Resolved block identity is `(device ID, original allocation event ID)`; + `block.id == allocEventId >= 0` stays stable across every slice containing it. + Reusing an address after a free creates a **different** block. Never deduplicate + by address or by a slice-local row number. +- Preexisting/unresolved blocks have stable, dataset/device-local negative + `block.id` values and `allocEventId=-1`. Repeated appearances keep their ID; + distinct objects at the same address must not collide. Negative identity is + not a known allocation event; a future enhanced identity/coverage table may + distinguish preexisting from other unresolved cases. +- `freeEventId=-1` means no observed free (live **or unknown**, not proof of a leak). + Otherwise it is a real ID in the device trace, not earlier than a known alloc. + `allocEventId=-1` means unknown/preexisting, not event 0. Other negative + lifecycle sentinels are invalid. Full finalized IDs/lifetimes may point outside + a slice's range. Slice observations may differ in state, but repeated IDs must + agree on address, size, requested size, alloc/free IDs. Finalization cannot + leave temporary placeholder IDs for a resolved allocation. + +The base format cannot prove that two colliding unresolved records are distinct +if a producer has discarded their identity; correct producer tracking is required. + +## Extensions, scopes and capabilities + +Keep optional metadata, deduplicated stacks and ordered frames in isolated +`pt_snap_*` extension tables or sidecars, never append columns to positional base +tables. In particular, **do not attach an unprefixed native `callstack` table to +inline v1**: the existing pt-snap layout detector correctly considers it a v1/v2 +conflict. Native v2 remains a separate single-DB layout, not this compatibility +base. `pt_snap_metadata` and auxiliary reference tables are optional here. + +Unknown `extensions` declarations/tables are ignored by base validation. +`DatasetValidation.text_callstacks=True` means an inline text column is available, +not that every event has a stack. `structured_frames=False` remains false even +when a manifest claims frames: complete structured-frame capability requires a +recognized extension version, ordered frame rows, valid references and coverage, +which P0 does not implement. Text stacks cannot substitute for these guarantees. + +`QueryScope.validate(manifest)` defines selectors, not execution support: + +| Scope | Required selectors and extent | +| --- | --- | +| `dataset` | No selectors; all declared devices and their full real traces | +| `device` | `device_id`; one declared device's full real trace | +| `slice` | `device_id`, `slice_index`; exactly that slice's real interval, boundary separate | +| `event_range` | `device_id`, inclusive `start_event_id/end_event_id` within the real trace, possibly across slices | + +There is no global event order across devices. Boundary IDs cannot be range +endpoints. P0 `DatasetValidation.query_execution=False` describes validation, +not execution. P1 accepts complete manifests/directories through the resolver +below; the explicit [P3 support matrix](querying.md#dataset-global-built-in-support-p3) +and [point-event sources](querying.md#dataset-point-event-attribution) define execution. + +```python +from pt_snap_cli.core.dataset_contract import QueryScope, validate_dataset + +inspection = validate_dataset("/capture/snapshot.pkl.msinsight") +QueryScope("event_range", device_id=0, start_event_id=1, end_event_id=3).validate( + inspection.manifest +) +assert not inspection.structured_frames +``` + +## Complete dataset focus and addressing (P1) + +A relocated compatibility-v1 directory or its `manifest.json` can be focused +without the original pickle, import metadata, or reference/extension tables: + +```bash +pt-snap focus /capture/snapshot.pkl.msinsight --device 0 --json +pt-snap overview --json +pt-snap query --template-use event --params '{"id": 3}' --json +pt-snap query --template-use event --slice 1 --json +``` + +`overview.dataset` reports format, manifest version/status, content fingerprint, +real/boundary counts, per-device slice paths/ranges and capabilities. Event bounds +exclude negative synthetic rows (also for standalone DB overview). Text stacks +are available, but not necessarily populated; P0 structured-frame validation remains +**unavailable** despite unknown extension claims. Cross-slice built-ins follow the P3 +matrix; recognized ordered-frame query coverage is separate from P0 validation. +External dataset-wide import metadata is unavailable; per-slice metadata alone +is not a source attestation. The compatible exporter below supplies a recognized, +whole-artifact `ptSnap` attestation, making its validated metadata available. + +Path precedence stays explicit → `PT_SNAP_DB_PATH` → nearest project focus → legacy +global config; device precedence stays explicit → focused → first discovered. +CLI focus only writes the chosen project/global focus file, never the artifact or +detected layout. API session focus does not write it: + +```python +from pathlib import Path +from pt_snap_cli import SnapshotAnalyzer + +with SnapshotAnalyzer(Path("/capture/snapshot.pkl.msinsight"), device_id=0) as analyzer: + event = analyzer.execute_query("event", {"id": 3}) + page = analyzer.execute_query("event", slice_index=1, exact_total=True) + print(event["scope"], page["total_is_exact"]) +``` + +The `event` template supports bounded addressing through dataset focus: a real ID +routes to its containing slice; inclusive `min_id/max_id` ranges may cross shards +with global filtering/sorting/paging; `--slice`/`slice_index` restricts real events. Negative IDs are not +whole-dataset selectors. Output `scope` identifies the actual DB/device/slice, +restricted real interval, dataset fingerprint and boundary exclusion. Totals and +pagination apply to this scope, **not** to an implicitly truncated full dataset. +Caller parameter defaults in CLI `effective_params` are additionally constrained +by `scope`. Point-event `active_blocks_at_event` and +`active_memory_callstack_at_event` now resolve alloc/free/full stack sources in +bounded batches before grouping/ranking, including original no-extension artifacts. +See [point-event attribution](querying.md#dataset-point-event-attribution) for +lifecycle identity, separate source/frame coverage, and the optional versioned +ordered-frame reader interface. Dataset-global built-ins use the explicit P3 matrix +and cumulative work ceilings; arbitrary SQL/custom overrides fail explicitly, never +silently choosing the first/latest slice. Single-DB real-event peak/gap and all-action +full-text stack-statistics versions are documented with the same contracts. + +`core.dataset_resolver.DatasetResolver.inspect(path)` returns an immutable +`ResolvedDataset` (or `None` for a standalone DB). Its `paths(QueryScope(...))` +addresses all matching files even for a cross-slice range, but does not execute it. +It revalidates the finalized P0 contract on each call and hashes the manifest and +members; no unbounded manifest/connection cache is retained. Hashing uses bounded +chunks and is proportional to artifact size, not a performance claim. Context LRU +size defaults to four; dataset generation changes invalidate contexts on next +lookup even when size/mtime are unchanged. Analyzer-owned caches close on exit; +injected caches remain caller-owned. Validation opens/closes slices sequentially. + +Building, missing/inconsistent slices, unknown base versions and unsafe paths +are rejected before focus writes or query execution. Before **any** SQLite open, +both the bare P0 validator and resolver check **all** member/parent aliases and +live/dangling WAL, journal and SHM sidecars. Finalized, closed compatibility-v1 +members use `mode=ro&immutable=1`, including cached query Context connections. +This accepts the fixed original producer's checkpointed, sidecar-free WAL headers +without creating sidecars or changing bytes. The immutable flag assumes a closed +finalized artifact, **not a live-WAL shortcut** or concurrent-writer guarantee. +No repair, checkpoint, journal-mode change, migration or deletion occurs. +Native datasets retain persistent-WAL rejection; standalone Context defaults stay +`mode=ro` without immutable. Cache identity includes transport mode and generation. +This is read-only validation, not a filesystem sandbox or a producer lock. + +Native-v2 private staging has **no published manifest**, so it is not a dataset; +individual DBs remain readable via the single-DB entry. Published native format +and compatible export are separate explicit protocols below. Cross-slice lifecycle +execution follows the P3 matrix; safe original-WAL reading is not GUI acceptance. + +## Published Native Dataset (P2) + +This is an **original, separately named and versioned** pt-snap protocol, not a +relabeled P0 artifact. `ImportService`/CLI can publish the internal native replay +result as `/.pt-snap-native-v2/`. The root must +contain exactly `manifest.json` plus the declared `device_` directories; each +device directory contains exactly its canonical `slice_.db` members. + +Required native manifest fields: + +| Field | Contract | +| --- | --- | +| `format` | Exactly `pt-snap-native-v2`, independently dispatched from compatibility-v1 | +| `schemaVersion`, `status` | Native manifest version `1`, only `complete` | +| `sourceFile` | Nonempty source basename matching `metadata.source_name`; never opened by analysis | +| `identity` | Exact fields `sourceSha256`, `requestedDevice`, `eventsPerSlice`, `outputFormat`, `manifestContractVersion`, `formatContractVersion`, `importContractVersion`, `extensionContractVersion`, `metadataSchemaVersion`, `importFormatVersion` | +| `metadata` | Complete existing `ImportMetadata` object, identical to the single row in every shard | +| `omittedDevices` | Distinct unselected/empty/static-only source device positions, disjoint from selected devices | +| `devices` | Canonical decimal keys; positive `eventCount`, `sliceCount`, all `readySlices` in order, ordered `slices` | +| Each slice | `index`, `startPosition`, `endPosition`, `startEventId`, `endEventId`, canonical relative `file`, `ready: true`, closed-member `sha256` | + +All semantic contract versions start at `1`; native metadata schema is `1` and DB +import format is `2`. `requestedDevice` is null (all event-bearing devices) or one +nonnegative integer. SHA256 strings are lowercase 64-digit hex. CLI +`metadata.importer_version` is informational, **not** a cache invalidation axis. +Native and future compatible identities are separately versioned; no native +`cacheHash` is generated or accepted as an identity substitute. + +Positions continuously cover `0..eventCount-1` per device, with every nonfinal +slice full and the final slice at most capacity. ID ranges are inclusive original +chronological bounds and may be sparse/nonzero; real count comes from positions +and actual rows, never `max(id)+1`. Negative synthetic rows are counted separately. +Each member has exactly its native-v2 trace/block device tables, `dictionary`, +`callstack`, `pt_snap_metadata`, and finalized `pt_snap_block_reference` v1. Native +trace columns match the standalone v2 schema (`callstackId`, not inline text), +with native OOM action 8 retained. Block IDs/lifetimes and text references must be +finalized, consistent across slices and refer to original IDs in that device. +Unknown extension declarations/tables do not grant structured-frame capability. + +The native resolver validates all member hashes, actual schemas/dictionaries, +real count/ranges, metadata and finalized references, retaining the P1 pre-open +alias/sidecar/persistent-WAL checks and bounded borrowed-cache ownership. Focus, +metadata, overview and single-slice `event` addressing work without the source +pickle. Native metadata is available dataset-wide; external compatibility metadata +remains unavailable unless the recognized compatible attestation below is present. +Native mode implies no migration, repair, compatible export, salted hash or GUI +acceptance. Dataset-global built-ins follow the P3 query matrix. Point-event source resolution is +available through the shared query path described above. P0 schema/manifest semantics are +unchanged; the compatibility-only finalized-WAL transport is described above. + +Publication and force/focus compensation are described in the +[import guide](quickstart.md#optional-import-a-native-sharded-dataset). A complete +manifest becomes visible only with all finalized members, never one ready shard. + +## Internal continuous replay (P1) + +`core.sharded_replay_service.ShardedReplayService.stage(source, directory, +events_per_slice=..., device=None)` is an **internal producer boundary**, not a +new CLI/API import command. It loads trusted pickle once, constructs one persistent +simulator per selected event-bearing device, and writes windows from last to first +into a new, caller-owned **private staging directory**. `omitted_devices` discloses +all unselected/empty device positions. Existing `import`, `split`, `SnapshotAnalyzer` +and single-DB v1/v2 reading are unchanged. + +The returned immutable result identifies **native-v2** slices, not compatibility-v1 +artifacts: trace rows use `callstackId` plus a local `callstack` table and retain +native OOM action 8 and workspace events. Sparse/nonzero-start native real IDs are +preserved if nonnegative, unique and chronological; windows count list positions, +not `max(id)+1`. These native slices are **not** accepted by `validate_dataset`. +Compatibility export, a compatible manifest, cache/publication and dataset queries +remain separate work. No `manifest.json`, `readySlices`, or incremental ready +notification is emitted. Only a successful return after all devices finalize and +validate may be handed to later publication. Failure retains private partial +files for their owner; none is advertised as a complete dataset. + +`SimulateDeviceSnapshot.replay_until(remaining_events)` undoes until that many +chronological list entries remain. The endpoint is a **position**, not an event ID; +repeated pauses/resumes share the full replay hook/error path. A real event row +records the state **after** its event (before undo). After undoing a window's first +event, active block observations and negative-ID `segment_alloc`/`segment_map` +rows reconstruct the state **before** that first event. Boundary rows do not count +against capacity or real-event peaks. + +A device-local registry retains original block objects across writer switches, +not copied hook payloads or an address-only key. Allocation replay resolves every +appearance to the original alloc ID; unknown/preexisting lifetimes retain stable +negative identities. All writers close before per-DB transactional batch backfill +and validation. `freeEventId` stays `free_completed`, never `free_requested`; +boundary `active_pending_free` observations remain state 0. Address reuse and +same-address different-stream blocks remain distinct. Each native slice adds +`pt_snap_block_reference(blockId, stream, allocCallstack, freeCallstack)` so finalized +out-of-slice lifecycle references do not discard their text stack sources. NULL +means no observed source event; empty text means the observed event had no frames. +Legacy slice-local query JOINs do not automatically consume this internal table; +this is not a cross-slice query engine or structured-frame extension. + +Pickle may still load entirely. Sharding bounds **real events per DB**, not block +rows, total staged bytes, registry size, or peak RSS. Synthetic regressions and +reviewed expandable/multi-device fixtures cover replay equivalence and resource +failure paths; no live upstream GUI or performance acceptance is claimed. + +## Explicit msinsight-Compatible Export (P2) + +`pt-snap import snapshot.pkl --format msinsight --json` (alias +`--format compatibility-v1`) targets **only** +`Ascend/msinsight@101f65b877a267ffd5f66ea3834706057ba243e5`. Its supported entrance is +opening the **same original pickle** in that target with adjacent +`.msinsight/`, not direct standalone-directory GUI import. +`--output-dir` selects another parent for pt-snap; relocation alone does not create +an upstream GUI entrance. `sourceFile` is the absolute original source path. +Capacity defaults to `500000` and can be set with `--events-per-slice`; default +standalone import and explicit native-v2 output remain separate and unchanged. + +The first-party converter consumes closed/finalized native staging, materializes +exact positional inline-v1 **tables**, removes unprefixed native `callstack`, and +writes physical-format metadata `1`. Original real IDs, actions, stream/metrics, +stable device-local block identities, `free_completed` and out-of-slice lifecycle +text references are preserved. No foreign real events are inserted to hide local +JOIN gaps. Unsupported OOM/unknown actions, nonzero/sparse IDs and empty/static-only +selections fail; native retains its wider contract. All selected members finalize, +close and validate before complete publication. `pt_snap_block_reference` v1 is an +isolated text extension, not ordered frames or a claim of cross-source queries. +Those queries and global aggregation follow the P3 query matrix. Text cannot recover +original ordered frame arrays; original workspace raw-frame loss and string-action +OOM coercion are known upstream limitations, not compatible features to imitate. + +`cacheHash = SHA256(b"mem_snapshot_parser_v2" + raw_pickle_bytes)`: the salt is +**prepended**. Independent `ptSnap.identity` fields are `sourceSha256`, +`requestedDevice`, `eventsPerSlice`, `outputFormat=compatibility-v1`, +`manifestContractVersion=1`, `exportContractVersion=1`, `referenceContractVersion=1`, +`metadataSchemaVersion=1`, `importFormatVersion=1`, and full `targetRevision`. +The `ptSnap` object also contains `metadata`, `omittedDevices`, and `members` +(canonical relative path → finalized SHA256). Whole validated ownership/identity +requires every member/hash and identical per-slice metadata; unknown/external +caches and unrelated files are not adopted. CLI version alone does not invalidate. + +Compatible publication is **no-replace even with force**. Identical recognized +pt-snap caches reuse without loading pickle (force does not bypass this rule). +Changed source path/content, options/contracts, corrupted members or GUI-modified +DBs require a new `--output-dir`; no repair, overwrite or deletion of the old +artifact occurs. Exclusive publication and focus compensation use the existing +checked transaction; source SHA256 is rechecked immediately before publish/reuse. +Native force replacement and recovery semantics are untouched. + +Inline text costs `sum(len(callstack.encode("utf-8")) for each event)` rather than +one copy per distinct stack. Existing native interning remains available. Conversion +temporarily holds both layouts and can leave free SQLite pages; capacity is not an +RSS/registry/block/disk bound. No optional frame enhancement is emitted. + +**GUI remains pending/not run.** Fixed-source parser checks complete manifest, +nonempty identical hash, every ready/resolved/existing member, read-only DB/device +validation and allocation-cache existence **or buildability** before UP_TO_DATE. +Derived allocation caches may modify DBs without invoking the producer: mtime or +new derived tables alone are not rerun evidence. Later GUI acceptance must retain +UP_TO_DATE/no-script-invocation logs and compare curves, blocks and cross-slice +presentation. Source checks and original-artifact production reads are not GUI, +C++ server/compile or full differential/performance acceptance. + +Immutable fixed blob evidence: +[parser predicates](https://api.github.com/repos/Ascend/msinsight/git/blobs/40046e6a92ff5af776ee55bd4cdd4586745af3c2), +[salt order](https://api.github.com/repos/Ascend/msinsight/git/blobs/ca421c7b33f3a0050df9e7a01e666f558801bc2f), +[allocation-cache predicates/build](https://api.github.com/repos/Ascend/msinsight/git/blobs/3d1ecc0ebd0d8e3eb0c984eda6b750ec4ce0e4f1). +The Ascend source is Mulan PSL v2, docs separately CC BY 4.0. This minimal converter +is original interface-facts code, not a copied vendor implementation or automatic +MIT relicensing of Ascend blobs. Existing pt-snap runtime lineage is a separate +license/provenance history. + +## Reproducible acceptance and measured limits + +[Acceptance and bounded performance](interop-acceptance.md) provides deterministic +CI, an optional hash-first existing-original consumer/physical-forward comparison, +the same-source three-mode local baseline and an explicitly pending GUI checklist. +These evidence layers stay separate; a raw negative token is only dataset-local, +never a cross-producer object-identity promise or permission to rewrite real IDs. + +## Evidence and limits + +Interoperability facts were read from fixed upstream +[manifest validation](https://github.com/Ascend/msinsight/blob/101f65b877a267ffd5f66ea3834706057ba243e5/server/src/modules/memsnapshot/service/MemSnapshotSliceService.cpp), +[positional SQLite reading](https://github.com/Ascend/msinsight/blob/101f65b877a267ffd5f66ea3834706057ba243e5/server/src/modules/memsnapshot/database/MemSnapshotDatabase.cpp), +[base schemas](https://api.github.com/repos/Ascend/msinsight/git/blobs/1fe50a5136f85365db55f4632d58f3cf7090431e), +and [publication/backfills](https://api.github.com/repos/Ascend/msinsight/git/blobs/95612509ef990ac6b7034919dae4938cc6acf3ad). +Those source files carry Huawei's Mulan PSL v2 notice. This contract, Python +validator and synthetic tests are original local implementation of interface +facts; no upstream implementation is copied into the package. The upstream docs +license is separately CC BY 4.0, not a replacement for those source notices. + +Our complete-readiness and canonical-path rules are intentionally stricter than +the upstream parser (which also supports building). Synthetic contract tests and +source-function comparisons are not GUI acceptance, native exporter acceptance, +performance measurements, or proof of interoperability with a different revision. diff --git a/docs/zh/database.md b/docs/zh/database.md index 94fdec2..4022e4b 100644 --- a/docs/zh/database.md +++ b/docs/zh/database.md @@ -11,6 +11,10 @@ SnapshotDB 是内存快照数据的 SQLite 数据库存储格式,用于持久 **数据库示例文件**: `snapshot.pkl.db` +分析目标包括单库与显式选择的完整已校验原生/兼容数据集;关闭的 compatibility-v1 +使用 immutable 只读传输,原生及单库策略仍独立。[验收、metadata/frame限制与测量](interop-acceptance.md) +区分 schema、来源/生命周期差分与待验收 GUI。 + ## 生成 SnapshotDB 如果输入是 PyTorch 原始 `.pkl` 内存快照,可以直接导入: @@ -368,3 +372,37 @@ CREATE TABLE pt_snap_metadata ( 旧版或外部生成的兼容数据库可以不包含此表,仍可正常查询;`pt-snap metadata` 会将其 metadata 状态报告为 unavailable。 + +### 已发布原生数据集 + +`import --events-per-slice N` 发布独立 `pt-snap-native-v2` manifest,不是兼容 v1 +manifest,也不改变单库布局。各最终化片使用已有 v2 表及 `pt_snap_block_reference` +(版本 1):`blockId INTEGER PRIMARY KEY`、`stream INTEGER NOT NULL`、 +`allocCallstack TEXT`、`freeCallstack TEXT`。它们记录文本生命周期来源,不是结构化 +frames 或自动跨片 JOIN。每片必需且一致的 `pt_snap_metadata` 之外,原生 manifest +身份另包含源/设备/容量、格式/manifest/导入/扩展合同版本,与 CLI 发布版本分离。 + +真实计数按各设备位置,保留原始稀疏 ID、OOM/workspace;负边界不占容量。 +reader 在有界 `event` 寻址前校验完整成员哈希、最终化身份/引用一致性,不修复、不迁移。 +数据集聚合/生命周期查询仍不支持。详见独立 +[原生 manifest 协议](sharded-snapshotdb.md#已发布原生数据集p2)及 +[事务恢复指南](quickstart.md#可选导入原生分库数据集)。 +P0 兼容 v1 表序、内联布局和 manifest 语义不变。 + +### 显式兼容导出 + +`import --format msinsight`(别名 `compatibility-v1`)生成物理 v1 +`trace_entry_` 表,第九列为 `callstack TEXT`,不是 VIEW;block/dictionary +位置 schema 精确匹配。转换去除暂存的未隔离 `callstack` 表,写入 +`pt_snap_metadata.import_format_version=1`,保留隔离的 `pt_snap_block_reference` +文本生命周期来源。原生单库/数据集仍使用去重和 v2 metadata,不生成或从文本推断有序 frames。 + +内联调用栈字节按**所有事件**的文本总和增长,不按不同栈计数;转换暂时并存原生与内联表, +可能保留 SQLite free pages。容量不限制 block、加载 RSS 或总磁盘。兼容 manifest 的 +`ptSnap.identity`、metadata、省略设备和全部成员哈希形成整体证明,完整校验后才报告 +metadata available;外部 v1 数据集仍不要求 metadata。 + +**仅最终化兼容数据集**的裸校验与缓存分析在首个 SQLite 打开前拒绝全部存活/悬空 +sidecar 和别名后使用 `mode=ro&immutable=1`,保留已关闭/checkpoint 原版 WAL 的字节及 +目录清单。前提是无活动 writer;原生数据集非 WAL 策略与单库 `mode=ro` 默认不变。详见 +[兼容协议及 no-replace 规则](sharded-snapshotdb.md#显式-msinsight-兼容导出p2)。 diff --git a/docs/zh/focus-management.md b/docs/zh/focus-management.md index e67d323..ba6689e 100644 --- a/docs/zh/focus-management.md +++ b/docs/zh/focus-management.md @@ -6,6 +6,11 @@ 该 focus,不同项目目录则可以选择不同数据库和设备。同一项目内的终端或 Agent 需要 相互隔离时,应使用 session 覆盖。 +目标也可为显式选择的 complete 已校验 compatibility-v1 / pt-snap-native-v2 目录或 +`manifest.json`,不能是任意目录/ready单片。明确选择设备,用 `overview.dataset` 记录 +format/fingerprint/真实区间。有效原版无扩展 metadata 为 `unavailable` / `metadata_missing` +(未知来源),不是绕过校验。只读诊断 skills 不持久化 focus;见[隔离验收项目](interop-acceptance.md)。 + ## 解析优先级 焦点按以下顺序解析: @@ -110,6 +115,36 @@ pt-snap config --json # 同一组操作用机器可读 JSON 输出 } ``` +## 完整数据集 focus 与导入补偿 + +完整兼容 v1 数据集或已发布 `pt-snap-native-v2` 目录/`manifest.json` 可以按相同的 +优先级和设备规则 focus: + +```bash +pt-snap focus captures/snapshot.pkl.pt-snap-native-v2 --device 0 --json +pt-snap query --template-use event --slice 0 --json +``` + +原生布局为 `v2`,兼容布局为 `v1`;写 focus 前校验全部声明成员,不修复产物,不保存 +布局信息。原生稀疏 ID 按原始 ID 寻址,不当成位置计数。数据集 focus 只支持有界 +`event`,聚合/生命周期及跨片请求明确失败。API 会话 focus 只读,`SnapshotAnalyzer` +不承担 import/split。 + +导入把项目 focus 写入**当前目录**,不是输出目录;`--no-focus` 跳过。原生数据集导入 focus 失败 +(包括先修改再抛错)在普通补偿下恢复旧产物与旧 focus 原始字节及内存 config 状态, +或者恢复原先不存在。恢复失败时,错误指明保留的 `.focus.pt-snap-*.recovery` checkpoint +及/或数据集恢复目录。若 checkpoint 为空且 `prior focus existed: False`,它记录原先 +不存在,不是有效 focus JSON。请保留证据供人工恢复;不承诺任意 I/O/崩溃原子性或并发 +writer 锁。详见[原生导入](quickstart.md#可选导入原生分库数据集)。 + +显式兼容导入(`--format msinsight`)采用相同的请求 focus 补偿,但即使 force 也 +**不替换**已有兼容输出。仅整体证明完整、身份相同的 pt-snap 缓存可复用,否则换新输出。 +GUI 仍要求同一原始 pickle 与邻接缓存,pt-snap focus 只需完整产物。首个 SQLite 打开 +之前检查全部兼容成员的别名及存活/悬空 sidecar;裸校验和缓存查询对已关闭最终化成员 +使用 `mode=ro&immutable=1`,保留原版 checkpoint WAL 的字节与目录清单。这不是 live +writer 锁,原生非 WAL 检查与单库默认不变。 + + ## 错误处理 **未设置焦点时查询:** diff --git a/docs/zh/interop-acceptance.md b/docs/zh/interop-acceptance.md new file mode 100644 index 0000000..a528cc5 --- /dev/null +++ b/docs/zh/interop-acceptance.md @@ -0,0 +1,157 @@ +# 双向互操作验收与有界性能 + +中文 | [English](../en/interop-acceptance.md) + +## 证据分层与固定目标 + +唯一上游目标是 **Ascend/msinsight@101f65b877a267ffd5f66ea3834706057ba243e5**。 +协议/schema、隔离源函数、合成 CI、真实原版生产端产物的读取差分、GUI/C++ 实测是 +不同证据层,不能互相替代。**GUI 已由人明确延期,待验收/not-run**;issue 保持开放。 +已有说明见 [#205](https://github.com/codeasier/pt-snap-cli/issues/205#issuecomment-5987264017),不重复发布。 + +验收工具是本地原创接口事实/独立参考实现,没有复制 vendor 实现。 +固定 [生产 schema](https://api.github.com/repos/Ascend/msinsight/git/blobs/1fe50a5136f85365db55f4632d58f3cf7090431e)、 +[manifest/回填](https://api.github.com/repos/Ascend/msinsight/git/blobs/95612509ef990ac6b7034919dae4938cc6acf3ad)、 +[parser 判据](https://api.github.com/repos/Ascend/msinsight/git/blobs/40046e6a92ff5af776ee55bd4cdd4586745af3c2)、 +[salt 顺序](https://api.github.com/repos/Ascend/msinsight/git/blobs/ca421c7b33f3a0050df9e7a01e666f558801bc2f) +为接口事实来源。Ascend 源码 Mulan PSL v2、上游 docs 单独 CC BY 4.0,不自动属于 +本项目 MIT lineage。真正移植须走 snapshot provenance/PR 声明并保留 notices。 +可执行 pickle 的信任是独立 fixture 审核;本验收不新增已提交 pickle。 + +## 正向:可信源 → 物理兼容导出 → 未来 GUI + +1. 先独立审核/信任源:pickle 可执行,不是沙箱。记录 SHA256、大小、设备与容量。 + 默认 import 仍单库,原生/兼容模式显式区分。 +2. 目标必须不存在。未来 GUI 支持入口要求产物邻接**同一原始 pickle**: + + ```bash + pt-snap import /capture/snapshot.pkl --format msinsight --events-per-slice 2000 --device 0 --no-focus --json + pt-snap overview /capture/snapshot.pkl.msinsight --json + ``` + + `--format compatibility-v1` 为别名;`--output-dir` 换父目录并不创建 GUI 直接导入目录 + 的入口。即使 force 也不覆盖原版/外部缓存,只复用已识别且身份完全相同的 pt-snap + 产物;成员变化使证明失效。`cacheHash=SHA256(b"mem_snapshot_parser_v2" + 原始字节)` + 与源/选项/格式/语义版本/成员构成的 `ptSnap.identity` 独立。 +3. 对同源、同选项的已有已审原版产物,逐片比较基础 schema/dictionary、全部九列真实 + event、ID/区间/action/stream/counter/文本及生命周期。非负 block ID **包括0**和七字段 + 必须完全相等。负 block ID 是生产端局部 token:全局及逐片比较其余六字段**包括 state** + 的 MULTISET,保留重复数量,并独立验证每份产物的 token 跨片稳定性。这不证明跨生产端 + 的负对象身份。不能按地址去重、改写 ID 或修补固定原版源码来制造等价。 +4. **将来另经人明确授权后**,使用记录的真实 GUI build 打开同一原始 pickle,补下方清单。 + 物理或 salted hash 相等都不等于 GUI 缓存复用通过。 + +## 反向:已有原版产物 → 完整 focus → canonical 参考 + +runner 不下载/执行生产端。原版产物必须已经存在、关闭且最终化,并有已审 receipt。 +允许没有 metadata/reference/frame 扩展:有效返回为 `unavailable` / `metadata_missing`, +只能记录未知来源,不能捏造 metadata。invalid、未完成校验及未知 reason 都停止。 + +```bash +pt-snap focus /capture/snapshot.pkl.msinsight/manifest.json --device 0 --json +pt-snap capabilities --json +pt-snap overview /capture/snapshot.pkl.msinsight --json +pt-snap query /capture/snapshot.pkl.msinsight --device 0 --template-use memory_peak --json +pt-snap report peak-memory /capture/snapshot.pkl.msinsight --device 0 --metric reserved --start-id 0 --end-id 100 --limit 20 --json +``` + +focus 持久化项目选择,应在明确批准的隔离验收项目使用。诊断 skills 本身不写 focus、 +不导入/安装。整份数据集 scope 只接受完整校验的目录/manifest,不接受任意目录或单个 ready 片。 +明确选择的单成员仍可按单库策略读取,但不是整份数据集 scope。 +已关闭兼容 v1 通过 immutable 只读 SQLite 接受 checkpoint 后无 sidecar 的 WAL header; +所有别名及存活/悬空 WAL/SHM/journal 在打开前拒绝。原生仍拒绝持久 WAL;单库仍 +`mode=ro`,不加 immutable。 + +在仓库根目录用预期解释器运行可选本地工具: + +```bash +PYTHONPATH=src PYTHONDONTWRITEBYTECODE=1 python benchmarks/interop_acceptance.py \ + --artifact /capture/snapshot.pkl.msinsight --receipt /evidence/reviewed-receipt.json \ + --source /capture/snapshot.pkl --device 0 --output /evidence/NEW-acceptance +``` + +可选 `--compatible /export/snapshot.pkl.msinsight` 比较**已有**正向产物,绝不自动生成。 +receipt 字段为完整固定 SHA `revision`、全部相对普通成员路径→SHA256 的 `artifactHashes`; +可选源/正向检查另要求 `fixtureSHA256`、`fixtureBytes`、`saltedCacheHash`。 +receipt 真实性依赖调用者审核,hash 本身不能证明生产端身份。源只计算 hash,不反序列化。 +在任何 SQLite/产品打开前及之后核对全部成员/hash/sidecar,成员变化失败。 +输出必须全新、规范、无符号链接且不与输入重叠;只在其中创建物理 canonical-reference DB、 +报告和隔离测试项目 focus,不改外部 focus/cache,不下载/执行生产端。 + +小型独立 oracle 显式物化有限数据(文件总32 MiB、256目录项、64片、含边界的20000条 +物理 trace、20000条 block、逐片256 dictionary 行、1 MiB JSON、可选2 MiB源)。这些是 +参考工具界限,不是产品 RSS 承诺。它比较全部真实列、完整 alloc/free_completed 来源、 +canonical 生命周期/最新状态、终片候选、最早 tie 的全局及逐片峰值、指定点活跃集合/字节、 +全部 action 栈统计与全局排序/分页/精确总数。canonical 单库不是又一次原版生产端执行。 +控制台链在切换隔离 CWD 前后验证同一解释器/相同 API 源路径,再运行真实 focus/query/report +并规范化 CLI/API scope/覆盖。报告记录实际选择的点/区间,不宣称枚举每个 event 的活跃集合。 + +## 确定性 CI 场景与限制 + +`pytest tests/test_interop_acceptance.py tests/test_dataset_import_baseline.py + tests/skills tests/test_bundled_skills.py` 使用小型生成 SQLite 或已有已审/合成运行时 fixture, +不依赖网络、GUI、NPU、live model 或私有绝对基线路径。任何已提交 pickle 运行前先执行 +`tests/test_fixture_provenance.py`。下列 owning suites 仍属于门禁: + +| 场景 | 确定性回归/口径 | +| --- | --- | +| 长寿命跨多片、地址复用、pending free、静态/预存/未知 | `tests/core/test_dataset_attribution.py`、`test_dataset_global.py`;按已证生命周期而非地址去重,free_completed 而非 free_requested | +| expandable、多设备、原生稀疏/非零 ID、OOM/workspace | `tests/snapshot/test_sharded_replay.py`、`tests/test_native_dataset.py`、`tests/core/test_msinsight_export.py`;原生保留,v1 明确拒绝 OOM/稀疏/非零 ID;原版 OOM 强制转换/workspace raw-frame 丢失不是通过功能 | +| 有序 frames | 只读 ptSnapOrderedFrames v1、实际 schema/覆盖、有限合法类型JSON/深度128/顺序/重复;缺失/未知/未覆盖→text-only,不生成/重建,不冒称原版生产 frames | +| 源/选项/格式/语义身份、成员变化、外部缓存 no-replace | 已有原生/兼容导入 suites 加验收测试;GUI 派生 DB 修改使证明失效,不证明重跑生产端 | +| 写库/回填/转换/源复核/发布/focus/rollback 失败 | `tests/core/test_dataset_import_failures.py`、`test_import_backend_failures.py`、`test_msinsight_export.py` 及分片回放;检查真实旧目标/focus/recovery 字节,不能只检查抛异常 | +| 全11内建、CLI/API/report、分页/预算 | 新物理参考与 `test_dataset_global.py`;自定义 SQL/同名 override 不支持,借用 LRU/终止关闭/handler 清理保持 | + +使用同一 YAML catalog 的 `dataset_support` 和声明的可选字段。区间 `scope.range_complete` +与定点 `scope.source_coverage.range_complete`、来源/frame 覆盖及 `has_more/truncated` 独立。 +数据集 canonical `source_stack_id` 不是局部 stack ID;代表行优先非空文本。缺引用、字面 +missing label、非NULL空文本活动组不能混同。`callstack_analysis` 语义 v2 的旧字段 +`alloc_count` 统计全部真实带栈 action,不是 alloc-only,`total_size` 为活动量而非存活字节。 +百分比分母保持过滤/top-N 后、caller 行上限前的 included group bytes(单库 SQL / 数据集 post-merge,#179),不使用全 active +计数器。报告 gap/归因/active coverage 来自同一 selected metric event。一次 query/report +共用递减时间及累计100000 fetched/output行、64 MiB序列化值,**不是 RSS**;失败不能给出 +部分的虚假全局结论。 + +## 性能:相同已审小样本,三模式 + +测量使用 `snapshot_expandable.pkl`,581140字节,SHA256 +`3afc9d1c5ef4ca4b417e58c0830e9eb8a913eb9459f4088b8c66c22325c68c40`,device0/capacity2000。 +旧 `baseline_import.py --samples 8k` 实际选择不同的多设备 fixture,不能当作同样本原版对照。 + +```bash +PYTHONPATH=src PYTHONDONTWRITEBYTECODE=1 python -m pytest tests/test_fixture_provenance.py +PYTHONPATH=src PYTHONDONTWRITEBYTECODE=1 python benchmarks/dataset_import_baseline.py \ + --source tests/fixtures/snapshots/snapshot_expandable.pkl \ + --sha256 3afc9d1c5ef4ca4b417e58c0830e9eb8a913eb9459f4088b8c66c22325c68c40 \ + --device 0 --capacity 2000 --output /evidence/NEW-three-mode --trusted-pickle +``` + +这是显式可信 pickle 执行同意,不是自动诊断步骤。不安装/重指 editable、不加载默认大样本、 +不重跑原版生产端、不 cleanup。每个 cold/reuse 使用新隔离子进程;cold 是全新输出、无warmup, +不是清空 OS page cache。reuse 必须是本次成功 cold 的未变所有产物,cache miss 禁止重建/覆盖; +同源/模式/phase所有权在 pickle 执行前检查。记录完整命令/退出/错误日志、source/head/script hash、 +Python/platform 和全部11查询延迟。[机器可读本地观测](../dataset-performance-baseline.json) 仅存 +去私有路径后的测量数据,不是速度断言或 CI 数值 oracle。 +同名支持查询的 scope 不一定相同:本次默认单库 `allocation` 返回8094行 +(8092真实+2合成边界),数据集全局只返回8092真实行;event定位选择显式真实ID。 +解读延迟/行数必须保留差异,不能因模板名称相同就宣称逻辑范围完全相等。 + +整次 import wall 与实际包装的 load/replay/SQLite/backfill/conversion **inclusive函数时间**不同。 +新子进程 RUSAGE_SELF 高水位在 import 结束、查询开始前采集,Darwin原始字节÷1024为KiB, +再÷1024为MiB,包含解释器/import开销。load/replay逐阶段RSS及未插桩剩余阶段时间明确 +**unavailable**并给原因,不能减累计高水位,也不能用父进程/上一个子进程的 RUSAGE_CHILDREN。 +总磁盘/逐片/重复block行/查询延迟实测,逐行物理page成本不可用。pickle仍整体加载,长寿命block +和兼容内联栈会重复,query hash/scan可能占主导。不承诺未测速度/RSS/磁盘收益。 +已有原版唯一冷wall0.160545s没有RSS/phase测量,不是受控比较性能基线。 + +## 未来 GUI 清单——全部待验收 / NOT-RUN + +- [ ] 精确真实version/build/commit/package SHA、解释器/server来源、环境、启动/open命令;不猜可执行语法。 +- [ ] 同一原始pickle路径/hash/size、邻接complete产物;打开前后全部manifest/member/hash清单,源不变。 +- [ ] UP_TO_DATE/cache-reuse日志、没有producer/parser脚本调用,并有足以发现重建的process/log观察;hash不足以证明。 +- [ ] 指定点的曲线/counter/range/真实ID(排除边界),block/size/state及跨片alloc/free_completed来源详情。 +- [ ] 仅实际recognized覆盖允许ordered frame声明,原版文本产物保持text-only;不可用展示如实记录。 +- [ ] 派生allocation cache修改单独检查:可能不运行生产端就改DB并使ptSnap证明失效;保留前后清单, + 不能修复/覆盖原版缓存来让reuse通过。 + +schema/合成/原版产物读取和静态skill评分不能勾选GUI项;本验收包不执行/编译GUI或C++。 diff --git a/docs/zh/querying.md b/docs/zh/querying.md index 6ad5ad8..69c243e 100644 --- a/docs/zh/querying.md +++ b/docs/zh/querying.md @@ -4,6 +4,12 @@ 对快照数据库执行内存分析查询。 +按同一 YAML catalog 的 `dataset_support`/声明字段选择显式有效 target/device/scope。 +[离线验收与实测查询延迟](interop-acceptance.md) 区分来源/区间/frame覆盖与 `has_more/truncated`, +说明 text-only降级、全部action栈统计并保持included-byte分母。有界 +`report peak-memory --start-id ... --end-id ...` 的 gap/active归因/coverage 使用同一 selected +metric event;整份数据集仍不支持自定义SQL/override。测量不是速度断言。 + ## Query 命令 ```bash @@ -16,7 +22,7 @@ pt-snap query [DB_PATH] [--template-use ] [--params ] \ | 参数 | 说明 | |------|------| -| `db_path` | SQLite 数据库文件路径(已配置 focus 时可选) | +| `db_path` | 单库 SQLite 文件或完整已校验数据集目录/manifest(已配置 focus 时可选) | | `--template-use` | 查询模板名称(除非使用 `--list` 或 `--template-info`,否则必需) | | `--params` | JSON 格式的查询参数 | | `--device` | 设备 ID | @@ -24,7 +30,7 @@ pt-snap query [DB_PATH] [--template-use ] [--params ] \ | `--category` | 按分类过滤模板:`basic`、`statistical`、`business` | | `--template-info` | 显示模板详情(参数、输出 schema 和字段语义) | | `-n` | 最大显示行数;零或负数表示不限制。与 `--timeout` 相互独立。 | -| `--exact-total` | 对匹配集合做 `COUNT`。默认 `total` 等于已返回行数。 | +| `--exact-total` | 计数完整匹配集合:单库 SQL COUNT 或数据集合并后的总数。默认 `total` 为返回行数。 | | `--timeout` | 一次 `query` / `QueryService` 调用的墙钟预算(页面查询与可选 `--exact-total` COUNT 共用),经 SQLite progress handler 生效。`<= 0` 表示关闭。默认:`PT_SNAP_QUERY_TIMEOUT` 或不限制。该环境变量作用于所有模板查询,包括 `report peak-memory`。 | | `--json` | 输出机器可读 JSON(执行、`--list` 与 `--template-info`) | @@ -71,6 +77,129 @@ pt-snap query '' --device 0 --template-use event --params '{"id":123}' 分组没有可取回的采集分配栈;在原查询中省略 `stack_bytes` 可查看完整显示标签。 查询保持只读,摘要不会改动数据库。 +## 数据集全局内建支持(P3) + +完整 native-v2 与 compatibility-v1 数据集共用以下**单设备**支持矩阵;catalog 与 +模板详情的 `dataset_support` 同步返回。设备优先级仍为显式 → focused → 首个发现; +不累加跨设备峰值,设备间没有共同全局事件顺序。 + +| 模板 | 最终输出窗口之前的数据集语义 | +| --- | --- | +| `event`, `allocation` | 筛选选定范围全部真实事件,按声明的 ID tie-break 全局排序,最后 offset/limit/max_rows;排除负边界 | +| `memory_peak` | allocated/active/reserved 独立真实峰值;同值选最早真实 ID;`start_id/end_id` 可跨片 | +| `allocator_gap` | 使用同样独立选择的全局事件,每个 gap 由**该事件**计数器计算 | +| `active_blocks_at_event`, `active_memory_callstack_at_event` | 路由实际事件到所有者片,分组/top_n 前解析完整跨片来源(见下文) | +| `preexisting_live` | 所有者事件的 preexisting 占用,不是泄漏数 | +| `block` | 已证明的分配事件生命周期及稳定负 token 去重后筛选/排序/分页;最新出现片 state 是观察,不是任意 E 的状态 | +| `freed_block_lifetime` | 已采集分配与已证明 `free_completed` 来源的去重生命周期;event-ID 距离桶,不是时间 | +| `leak_detection` | **数据集终片**中仍存在且无完成释放的已采集动态生命周期,不拼接逐片未释放列表;存活仅为候选,不证明泄漏;拒绝 `--slice` | +| `callstack_analysis` | 全部带栈真实 trace action(含 free/segment/workspace),完整 canonical 来源身份合并后再做全局阈值/排名;加权 `avg_size=total_size/alloc_count` | +| runtime override/自定义模板/任意 SQL | 数据集明确不支持,即使命名为内建模板;需显式传单库 member 使用局部 SQL | + +`memory_peak` 与 `allocator_gap` 语义版本 **2** 在**单库** SQL 中也排除负合成行。 +`callstack_analysis` 版本 **2** 保留旧 `alloc_count` 字段名,但**不是分配次数**: +v1 统计非 NULL 文本的全部真实行;v2 统计非 NULL 栈引用的全部真实行,缺失/NULL +joined 文本保留为独立 missing 组。空文本单独计数组,与 missing 及真实字面量标签分开; +两种单库布局都按完整文本分组而非局部 v2 ID。数据集还接受明确覆盖的有序原始数组 +作为栈证据;缺失文本不改变原始数组 +身份。size 总和含非分配 action,不是瞬时活跃占用。 + +共享 YAML 输出 schema 声明全部可选的仅数据集行字段;单库行形状不变。`block` 与 +数据集 `leak_detection` 提供 `lifecycle_id`、身份/来源证明状态、来源载荷、最新观察片/ +state scope、category 与终片存活;leak 行另保留 `requestedSize`、`state`、`freeEventId`。 +这些观察与证明不等于确认泄漏。数据集统计提供 `source_stack_id`、`stack_kind`、 +`text_kind`、`stack_event_id`、`frames_status`。真实代表事件优先选可用的非空采集文本, +但不是 canonical 分组身份,也不一定是 allocation action;无采集文本时可指向空/NULL +文本。`text_kind` 仅为 provenance,不改变原始数组身份。在同一数据集/设备用 `event` +取回真实文本/数组;`frames_status` 不表示从文本重建 frame。 + +```bash +pt-snap query '' --device 0 --template-use memory_peak --params '{"start_id":100,"end_id":700}' --json +pt-snap query '' --device 0 --template-use event --params '{"min_id":100,"max_id":700,"limit":20,"offset":20}' --exact-total --json +pt-snap query '' --device 0 --template-use leak_detection -n 20 --exact-total --json +pt-snap report peak-memory '' --device 0 --metric reserved --start-id 100 --end-id 700 --timeout 10 --json +``` + +请使用实际范围。区间可包含稀疏缺口,不虚构事件;无真实事件的范围返回 NULL 峰值/ID。 +越界/负/反向端点失败;`event(id=gap)` 可为空,active-at-gap 失败。全局筛选、去重、来源 +身份、阈值、排序都在最终窗口之前。`exact_total` 忽略行数上限;默认 total 为返回行数, +exactness 标记诚实。`has_more/truncated` 描述窗口,不表示来源/范围完整性。`scope` 返回 +请求区间、适用时实际真实事件覆盖、涉及 `slice_indices`、设备、指纹及边界排除;来源覆盖 +独立。未证明历史身份按片局部单独表达,不按地址/局部栈 ID 猜合并,不断言为已证明泄漏。 + +数据集读取先分离批次再查外片来源,使用借用的有界 Context LRU。所有片、来源、分组、 +精确总数共用递减 deadline;报告三条查询明确共用**一个**预算。累计取回/输出工作上限为 +**100000 行 / 64 MiB 序列化值**,含重复来源读取;超限整体失败,不返回冒充完整的部分 +结论。物化/排序由工作上限约束,**不由 max_rows 约束**;不是进程 RSS 上限,Python/ +SQLite、一个批次及单个 cell 有额外开销。manifest hash/校验仍与产物大小成正比且仅在 +阶段边界检查,不硬抢占。不生成合并临时 DB、不修复或写产物。 + +## 数据集定点事件归因 + +完整兼容 v1 或原生 v2 目录/manifest 支持 `event` 寻址,以及接受真实 `event_id` 的 +`active_blocks_at_event`、`active_memory_callstack_at_event`。后两者按设备路由实际事件, +从所在片完整生命周期观察计算正向事件**之后**的集合:`allocEventId <= E` 且完成释放 +`freeEventId > E`(或 `-1`)。`free_requested` 不移除 active 内存。负边界行表示片内 +首事件之前,不是定点端点。原生稀疏 ID 不改号:片内 `event(id=gap)` 可返回空行, +不存在事件的 active 归因则明确失败,不虚构 event 0。 + +```bash +pt-snap query '' --device 0 --template-use active_memory_callstack_at_event --params '{"event_id":700,"top_n":20}' --json +pt-snap query '' --device 0 --template-use active_blocks_at_event --params '{"event_id":700,"limit":10}' --exact-total --json +``` + +请替换实际路径/设备/事件。片外 alloc/free 事件及完整 v1 内联或 v2 局部 ID 栈在 +分组/top_n **之前**解析,按实际来源片每批至多 256 个不同事件 ID。不补入外片真实行。 +原版产物无需 metadata 或引用表。生命周期用数据集 generation/设备/分配事件或稳定 +负身份,不用地址;`state_scope=slice_observation` 提醒存储 `state` 不一定是 E 时的 +pending-free 状态。block 行提供 `allocation_source`、`free_source`、状态和 +`lifecycle_id`。缺失动态来源仍保留字节及动态类别,与 static/preexisting 未知历史分开; +`free=-1` 表示存活**或未知**,不证明泄漏。 + +数据集分组始终提供完整 `source_stack_id`/`stack_id`、`stack_kind`、`stack_event_id`。 +无扩展的默认 `event` 行保留原始九个值;`stack_bytes>=0` 或已识别有序 frame 声明时 +才增加来源/frame 行字段,`scope.source_coverage` 仍返回。 +两种数据集布局均按完整文本的 canonical 身份分组;v2 仍按来源片真实局部 ID 查栈, +来源 `local_stack_id`/`slice_index` 是 provenance,不是跨片分组 key。有覆盖的有序数组 +只使用精确原始 frame 身份,与格式化文本是否存在无关;`text_kind` 独立描述文本 +可用性。明确覆盖的数组(含空数组)是已采集原始证据;分组优先选择有文本的代表 +事件以便取回全文。非空 whitespace 和真实采集的字面量 +`[missing callstack]` 不是 missing。禁止按局部 ID 或缩略显示标签合并。`stack_bytes` +只缩短显示;在同一数据集/设备用 `event(id=stack_event_id)` 取回全文。明确覆盖的原始 +有序 frames 也参与身份;不从文本重建 frames。 + +`scope.source_coverage` 独立于 `has_more`/`truncated`/精确行数:active 查询覆盖的是 +**top_n/分页之前全部筛选 block**,包括 active/dynamic 字节、已解析 alloc/free 数、采集 +动态字节、未知/预存数、有序 frame 覆盖及降级。`event` 覆盖仅为 `returned_events`。 +完整 manifest/范围不证明历史来源或 frame 完整。百分比分母仍为动态 top_n **之后、 +max_rows 之前**所含字节,不是数据集 active 计数器。精确总数忽略行数/排名上限,但 +不改变百分比分母。 + +一次调用的来源批次、分组和精确总数共用 deadline。manifest 校验/hash 及 Context +设置耗时计入预算,但同步文件 I/O/校验只在阶段边界检查,不硬抢占,不保证 OS 级 +时间上限。SQL progress handler 退出即清除。Context LRU 有界(默认四个),借用 +cache 仍由调用者所有。不生成合并临时数据库。`ReportService.event_attribution(E, ...)` +对给定事件复用同一路径。数据集全局查询及报告遵循上文明确支持矩阵;任意 SQL/ +自定义 override 不会因使用内建名字就继承全局合并语义。 + +### 可选有序 frame reader 合同 + +这是新增的仅 reader pt-snap 扩展,不是原版 producer 已有功能。当前原生/兼容 writer +**不生成**有序 frame 表。manifest 声明 `extensions.ptSnapOrderedFrames={"version":1}`, +并在有覆盖的来源片使用隔离的实际表: + +- `pt_snap_frame_coverage(eventId INTEGER PRIMARY KEY, frameCount INTEGER)` +- `pt_snap_frame(eventId INTEGER, frameIndex INTEGER, frameJson TEXT, PRIMARY KEY(eventId,frameIndex))` + +原始 object 必须是有限数字的标准 JSON,嵌套深度至多 128;超深、非有限数或解析 +失败(包括递归错误)安全降级。每个覆盖的真实事件须有非负整数 `frameCount`、恰好 `0..frameCount-1` 索引及保留全部 +原始类型字段的 JSON **object**。保持顺序和重复帧;空数组须有明确 count=0 行。 +只有已识别版本/schema 及有效逐事件覆盖才输出 `frames`、`frames_status=ordered`。 +未知版本、缺失 schema/行、无效 JSON 或覆盖不全降级为 `frames=null`、 +`frames_status=text_only`;不会切分/反转格式化文本伪造原始 frames。P0/overview 的 +数据集级 `structured_frames=false` 仍是保守校验能力,不表示定点来源载荷必然无 frames。 +本接口不提供树渲染或 GUI 验收。 + ## 查询模板 模板分为三个分类。用 `pt-snap capabilities --json` 一次查看完整清单(CLI 版本、全部模板契约和随包 skill),或用 `pt-snap query --list` 查看名称与描述。用 `--category` 过滤。 @@ -250,7 +379,7 @@ pt-snap query --template-use allocator_gap 如果需要更高层的摘要,可以使用: ```bash -pt-snap report peak-memory [db_path] [--device ] [--metric active|allocated|reserved] [--include-static|--exclude-static] [--limit ] [--json] +pt-snap report peak-memory [db_path] [--device ] [--metric active|allocated|reserved] [--include-static|--exclude-static] [--limit ] [--start-id ] [--end-id ] [--timeout ] [--json] ``` 示例: @@ -272,7 +401,10 @@ pt-snap report peak-memory /path/to/snapshot.db --json - `allocator_gap` - `active_memory_callstack_at_event` -并输出人类可读摘要或 JSON。 +并输出人类可读摘要或 JSON。`--start-id/--end-id` 限制峰值选择;归因仍解析所选事件 +完整活跃集合,包含更早分配。`--timeout`(或 `PT_SNAP_QUERY_TIMEOUT`)是递减的 +**报告整体** deadline,不是重置三次的逐查询超时。JSON 新增峰值 `scope`、独立归因 +`source_coverage`、`timeout_s` 与 `budget_scope="report_composition"`。 原有 `callstack_groups` 和 `percent_of_active_blocks` 数值保持不变。JSON 还会 返回归因的 `has_more`、`truncated`、`total_is_exact` 和 `effective_params` @@ -305,8 +437,9 @@ pt-snap report peak-memory /path/to/snapshot.db --json `-n` 仍然限制 `rows` 与 `returned`。默认 `total` 等于 `returned`(本页行数); 仅当本页就是完整匹配集合(`has_more` 为 false 且 `offset` 为 0)时 -`total_is_exact` 为 true。`--exact-total` 会对匹配集合做 `COUNT`(忽略 -`limit` / `offset` / `top_n`)并把 `total_is_exact` 设为 true。存在有限 +`total_is_exact` 为 true。`--exact-total` 计数匹配集合(忽略 `limit` / `offset` / +`top_n`)并设置 exactness;单库使用 SQL `COUNT`,数据集内建查询计数完整合并/去重后、 +全局窗口前的匹配集合,不计数已截断的局部片行。单库路径存在有限 尾部 `LIMIT` 时,执行器会多取一行来设置 `has_more`,不必先做计数。CTE 内部的有限 `top_n` 不是尾部 `LIMIT`:排名窗口已满时会置 `has_more` / `truncated`,避免把默认页报成完整集合;精确 `COUNT` 若等于 `returned` @@ -317,8 +450,9 @@ QueryService 调用的共享预算(页面查询与可选 COUNT 共用),作 模板查询(含 `report peak-memory`),不改变行数上限。非数字的 `PT_SNAP_QUERY_TIMEOUT` 归为 `INVALID_PARAMETER`,不是 `QUERY_FAILED`。 -`effective_params` 是应用默认值并按 `choices` 规范化后的参数。带 `-n` 时, -`limit` 是与执行器相同的**尾部** SQL `LIMIT`:已声明的模板 `limit` 与 `-n` +`effective_params` 是应用默认值并按 `choices` 规范化后的参数。数据集 `limit` 表示 +最终全局输出窗口上限(相同的正上限取小规则),不是逐片 SQL LIMIT;筛选/合并已经完成。 +单库路径带 `-n` 时,`limit` 是与执行器相同的**尾部** SQL `LIMIT`:已声明的模板 `limit` 与 `-n` 取较小值;渲染后的 SQL 没有尾部 `LIMIT` 时追加 `-n`。CTE 内部的 `top_n` 仍是独立参数,不会改写 `limit`。 `--template-info --json` 包含 `semantics_version`、`interpretation_limits` diff --git a/docs/zh/quickstart.md b/docs/zh/quickstart.md index 9a82241..c04e582 100644 --- a/docs/zh/quickstart.md +++ b/docs/zh/quickstart.md @@ -4,6 +4,10 @@ 几分钟内即可上手 `pt-snap-cli`。 +显式原生/兼容导入验收、相同已审 fixture 三模式性能及待验收 GUI 清单见 +[双向互操作验收](interop-acceptance.md)。默认导入仍是单库,`--events-per-slice` +选择原生,`--format msinsight` 选择 compatibility-v1;可信导入同意与只读诊断分开。 + ## 安装 ```bash @@ -44,6 +48,79 @@ SnapshotDB 会记录每个事件后的 `allocated`、`active`、`reserved` 总 生命周期,供 `pt-snap query` 与 `pt-snap report` 使用。回放是命令工作流的一部分; 快照运行时的 Python 模块不是公开 API。 +### 可选:导入原生分库数据集 + +```bash +pt-snap import snapshot.pkl --events-per-slice 50000 --output-dir captures --json +pt-snap overview captures/snapshot.pkl.pt-snap-native-v2 --json +pt-snap query --template-use event --slice 0 --json +``` + +不指定 `--events-per-slice` 时仍生成已有单 DB;指定后生成 +`/<完整输入文件名>.pt-snap-native-v2/`,省略 `--output-dir` 则放在源文件旁。 +此模式可显式指定 `--format pt-snap-native-v2`;`--format single-db` 不允许容量参数。 +原生产物**不是** msinsight 兼容导出;显式 `--format msinsight`(别名 +`compatibility-v1`)选择下述独立兼容模式。 + +容量按**各设备**的真实时间顺序位置计数,不使用 `max(id)+1`,不计负 ID 合成边界。 +保留原始稀疏/非零 ID、OOM 和 workspace 事件。`--device` 选择一个有事件设备,默认 +导入全部有事件设备;空/仅静态及未选设备位置由 `omitted_devices` 披露。选择空设备或 +全部为空的输入会失败。分库不限制 pickle 加载、registry/block 行数或峰值 RSS。 + +缓存复用须校验完整 manifest 与**全部**已关闭成员,并匹配源 SHA256、设备选择、容量、 +原生格式、manifest、导入语义、扩展及 metadata/布局合同版本。仅 CLI 发布版本变化不会 +重建;原生身份与后续兼容格式及 msinsight salted `cacheHash` 分离。JSON 增加 +`dataset_path`、`devices`、`slice_count`、`format`、`omitted_devices`、`reused` 和 +`rebuilt`;此模式下 `db_path` 为数据集目录。 + +匹配目标可复用;不同或成员损坏的已识别原生数据集会保留,除非显式 `--force` 替换。 +未知目录、无关额外文件、非法 manifest 和符号链接目标即使 force 也不会被接管或删除, +请换一个 `--output-dir`。所有设备/分片在目标同级暂存,完成回填、关闭、metadata 写入与 +校验,并在发布前复核源哈希;部分就绪片不构成缓存。 + +新目标使用排他的目录 rename 发布。force 先把旧产物移入所有权受检的恢复目录,再发布 +新产物;**路径有短暂空档,不是单次原子 swap**。请求的项目 focus 纳入补偿,写入已修改 +文件后才抛错也要处理:普通失败恢复旧产物及旧 focus 原始字节/状态(或原先不存在)。 +`--no-focus` 完全跳过 focus。回滚自身失败会保留并明确报告恢复路径,检查证据前不要删除。 +不保证任意 I/O 故障、并发 writer 或崩溃下无条件原子恢复。 + +数据集 focus 支持有界 `event` 寻址及共用跨片来源的定点 active block/调用栈归因。 +数据集全局 peak/list/leak 聚合仍明确失败,不按一片近似。详见 +[定点覆盖](querying.md#数据集定点事件归因)及[原生 manifest 合同](sharded-snapshotdb.md#已发布原生数据集p2)。 +`SnapshotAnalyzer` 仍只负责分析,不新增 import/split 方法。 + +### 可选:导出 msinsight 兼容数据集 + +```bash +pt-snap import snapshot.pkl --format msinsight --json +pt-snap overview snapshot.pkl.msinsight --json +pt-snap query --template-use event --slice 0 --json +``` + +此**显式**模式仅对齐 `Ascend/msinsight@101f65b877a267ffd5f66ea3834706057ba243e5` +的原始 pickle 导入入口,不是独立目录入口。GUI 发现要求同一原始 pickle 邻接 +`<完整输入文件名>.msinsight/`;pt-snap 可在没有 pickle 时分析已搬迁完整产物。 +可选 `--events-per-slice N` 默认 `500000`;`--output-dir` 选择其他父目录,不意味着 +GUI 支持独立入口。`--device` 选择有事件设备,并披露省略设备。非零/稀疏真实 ID、 +OOM/未知 action、空/仅静态选择明确失败,不静默映射;原生模式仍保留稀疏 ID、OOM、 +workspace。可信 pickle 与完整加载的内存成本仍适用。 + +基础表是物理内联 v1 表,metadata import format 为 `1`。调用栈文本按**每个事件** +复制,不按不同栈只存一次,磁盘与暂存转换空间可能明显增加;原生去重不变。 +不从格式化文本虚构有序结构化 frames。数据集可用有界 `event` 和定点 active +block/调用栈查询;全局聚合/报告事件选择仍不支持。 + +兼容发布采用 **no-replace**:外部/原版/用户缓存即使 `--force` 也不覆盖、不删除。 +仅完整、已证明且身份相同的 pt-snap 缓存可复用(force 也复用);否则换新 +`--output-dir`。源内容、sourceFile、选项、全部成员哈希和合同版本须独立于 salted +`cacheHash` 匹配。GUI 修改派生缓存可能使该证明失效,但不等于重跑生产者。 +全部暂存成员关闭/最终化/校验后才排他发布并写请求的 focus;普通失败补偿旧 focus +原始字节,回滚失败则保留恢复证据。单库/原生 force 策略不变,`SnapshotAnalyzer` 只分析。 + +**GUI 待验收/未运行。** complete/ready、安全路径/设备表及派生 allocation cache +存在或可构建同样重要;hash 相同不证明 GUI 复用或显示一致。详见 +[兼容协议及固定源码证据](sharded-snapshotdb.md#显式-msinsight-兼容导出p2)。 + ### 可选:拆分快照 如需生成更小、可独立回放的文件,可使用 `pt-snap split`。该命令不会读取或修改 focus: diff --git a/docs/zh/sharded-snapshotdb.md b/docs/zh/sharded-snapshotdb.md new file mode 100644 index 0000000..137d3fd --- /dev/null +++ b/docs/zh/sharded-snapshotdb.md @@ -0,0 +1,368 @@ +# 分片 SnapshotDB 协议(兼容 v1 与原生 v2) + +中文 | [English](../en/sharded-snapshotdb.md) + +## 范围与版本 + +本文定义产物协议、共享校验模型及下述**显式**兼容 CLI 导出器;跨片执行由 +[P3 内建支持矩阵](querying.md#数据集全局内建支持p3) 单独定义。 +`import` 默认仍生成一个原生 v2 DB;`split` 仍输出可回放 pickle/JSON。现有单库 +v1/v2 分析入口保留;P3 真实事件 peak/gap 与全部 action 完整文本栈统计语义在查询指南明确版本化。下述 P1 reader 新增将 complete 兼容 v1 manifest 或目录作为 +focus;`complete` 门槛只约束数据集,不约束旧单库分析。 + +`pt_snap_cli.core.dataset_contract` 提供 `parse_manifest`、`validate_event_ids`、 +`validate_dataset`、不可变 manifest/device/slice 模型、`QueryScope` 和带 +`location/code` 的 `DatasetContractError`。生产端可在发布前与读取端共用同一合同。 +校验不加载 pickle、不创建缺失数据库、不迁移 SQLite、不写 focus。 + +版本轴分别管理: + +- `schemaVersion=1`:本文的 manifest 和兼容基础表协议。 +- `pt_snap_metadata.import_format_version`:pt-snap 导入格式,原生为 `2`,兼容内联为 `1`; + **不是** manifest 版本,外部产物不要求该表。 +- 后续结构化 frame 扩展独立版本化。未知扩展不影响基础读取,也不意味着支持结构化 frames。 + +## 目录与 manifest + +生产端布局为 `<原始 snapshot 路径>.msinsight/manifest.json`,分片相对于该目录放在 +`device_/slice_.db`。允许显式指定已搬迁的产物目录进行校验; +`sourceFile` 记录来源,不是要打开的路径。检查时不得加载源 pickle。 + +```json +{ + "schemaVersion": 1, + "status": "complete", + "sourceFile": "/capture/snapshot.pkl", + "cacheHash": "producer-specific-opaque-identity", + "eventsPerSlice": 2, + "devices": { + "0": { + "eventCount": 4, + "sliceCount": 2, + "readySlices": [0, 1], + "slices": [ + {"index": 0, "startEventId": 0, "endEventId": 1, "file": "device_0/slice_00000.db", "ready": true}, + {"index": 1, "startEventId": 2, "endEventId": 3, "file": "device_0/slice_00001.db", "ready": true} + ] + } + } +} +``` + +示例中所有字段必需。整数不能是布尔或浮点数;计数/区间使用有符号 64 位整数, +版本/index/sliceCount/device ID 使用非负有符号 32 位整数。设备 key 为规范十进制 +ID(`0` 而非 `00`)。`sourceFile` 非空。`cacheHash` 是生产端不透明 token, +外部产物可为空:**不能**假定它是 SHA-256 或源内容证据。未来缓存需要区分源内容、 +生产端/格式、基础/扩展版本、所选设备、容量和最终化 generation;单独的源路径、 +size/mtime、`cacheHash` 或单片 index 都不足以复用。本 P0 模块不执行缓存复用。 + +`building` 只允许用 `parse_manifest(..., require_complete=False)` 检查部分就绪状态, +仍须声明完整计划区间。`complete` 必须在**身份/生命周期回填结束后**使所有片就绪。 +就绪 index 不重复、不越界,并与各片布尔值一致。`validate_dataset` 只接受所有文件 +齐全的 complete 数据集;不会创建缺片。重复 JSON 成员被拒绝。 + +各设备分片从 index 0 顺序排列;闭区间无重叠/缺口地连续覆盖 `0..eventCount-1`。 +每片最多 `eventsPerSlice` 个真实事件(允许中间片较短)。按真实事件位置计数,不能 +用 `max(id)+1` 推算。兼容 v1 **拒绝非零起点、稀疏、乱序或重复的原始真实 ID**, +不得静默改号。空设备和仅有静态 block 的设备无法表示:请求此类设备时明确拒绝, +不得虚构 event 0;全部为空的数据集被拒绝。可明确选择其他设备,但生产端必须披露省略。 + +路径须严格对应声明的 device/index 相对路径;拒绝绝对/盘符/UNC 路径、`..`、`.`、 +空分量、反斜线、NUL、设备错配、别名及成员/父目录符号链接。校验以 `mode=ro` +打开 SQLite,包括失败路径都关闭资源。这不是 OS 沙箱,也不锁住并发发布。 + +## 固定基础 SQLite schema + +每片须有本设备的**实际表**,不能用兼容 VIEW 替代:`trace_entry_`、`block_`、 +`dictionary`。上游部分 `SELECT *` 按位置读取,因此列序属于合同。原生 v2 的 +`callstackId` 不能替代内联文本。 + +| 表 | 按顺序排列的列(SQLite 声明类型) | +| --- | --- | +| `trace_entry_` | `id INTEGER PRIMARY KEY`, `action INTEGER`, `address INTEGER`, `size INTEGER`, `stream INTEGER`, `allocated INTEGER`, `active INTEGER`, `reserved INTEGER`, `callstack TEXT` | +| `block_` | `id INTEGER PRIMARY KEY`, `address INTEGER`, `size INTEGER`, `requestedSize INTEGER`, `state INTEGER`, `allocEventId INTEGER`, `freeEventId INTEGER` | +| `dictionary` | `table TEXT`, `column TEXT`, `key TEXT`, `value TEXT` | + +dictionary 的十进制整数 key 以文本存储,使用真实带设备后缀的表名与列名。每片包含 +以下完整且精确的映射: + +| 列 | key → value | +| --- | --- | +| `trace_entry_.action` | `0 → segment_map`, `1 → segment_unmap`, `2 → segment_alloc`, `3 → segment_free`, `4 → alloc`, `5 → free_requested`, `6 → free_completed`, `7 → workspace_snapshot` | +| `block_.state` | `-1 → inactive`, `0 → active_pending_free`, `1 → active_allocated` | + +不接受未知基础 action/state。特别是原生 pt-snap 的 `oom=8` 不在固定版本兼容基线内; +未来导出器须拒绝或协商独立扩展,不得重标记。指标/大小单位为字节;event ID 是按时间 +排序的身份,**不是时间戳**。 + +## 边界事件与 block 生命周期 + +- 真实事件保留非负原始 ID。`eventCount`、容量、范围及真实事件峰值只计数/筛选 `id >= 0`。 +- 负 trace ID 独立标识合成的左边界状态行(`segment_map=0` 或 `segment_alloc=2`)。 + 即使内存总量很大,也不能竞争真实峰值或占容量。ID 在片内局部,使用 + `(device, slice, 负 ID)` 标识边界行;重复边界 segment 不等于新真实分配。 + 校验分别报告 `real_event_count/boundary_event_count`,不执行峰值查询。 +- 已解析 block 的身份是 `(device ID, 原始 allocation event ID)`;所有出现片中 + `block.id == allocEventId >= 0`。释放后同地址复用产生**另一个** block。 + 禁止按地址或片内行号去重。 +- 预存/未解析 block 使用数据集/设备内稳定的负 `block.id`,`allocEventId=-1`。 + 重复出现保留 ID,同地址不同对象不得碰撞。负身份不是已知分配事件;未来增强身份/覆盖表 + 可区分预存和其他未解析情况。 +- `freeEventId=-1` 表示没有观察到释放(存活**或未知**,不证明泄漏);否则必须是设备内 + 真实 ID,不能早于已知 alloc。`allocEventId=-1` 表示未知/预存,不是 event 0; + 其他负生命周期哨兵非法。最终化的完整 ID/生命周期可指向当前片范围外。各片状态观察可 + 不同,但重复身份的地址、大小、requested size、alloc/free ID 必须一致。已解析分配 + 最终化后不得遗留临时占位 ID。 + +若生产端已丢失身份,基础格式无法证明两个碰撞的未解析记录是不同对象;生产端必须正确跟踪。 + +## 扩展、scope 与能力 + +可选 metadata、去重 stack、有序 frame 使用隔离的 `pt_snap_*` 扩展表或 sidecar, +不能向按位置读取的基础表追加列。尤其**不能给内联 v1 附加未命名隔离的原生 +`callstack` 表**:现有布局检测会正确报告 v1/v2 冲突。原生 v2 仍是独立单库布局, +不是本兼容基础格式。这里不要求 `pt_snap_metadata` 或辅助引用表。 + +基础校验忽略未知 `extensions` 声明/表。`DatasetValidation.text_callstacks=True` +表示有内联文本列,不保证每个事件有栈。即使 manifest 声称 frames, +`structured_frames=False` 仍为假:完整结构化 frame 能力要求已识别的独立扩展版本、 +有序 frame 行、合法引用及覆盖率,本 P0 不实现。文本栈不能代替这些保证。 + +`QueryScope.validate(manifest)` 只定义 selector,不宣称支持执行: + +| scope | 必需 selector 与范围 | +| --- | --- | +| `dataset` | 无 selector;所有声明设备的完整真实 trace | +| `device` | `device_id`;一个声明设备的完整真实 trace | +| `slice` | `device_id`, `slice_index`;该片真实区间,边界独立 | +| `event_range` | `device_id`、设备内闭区间 `start_event_id/end_event_id`;可跨片 | + +设备间没有全局事件顺序。边界 ID 不能作为范围端点。P0 的 +`DatasetValidation.query_execution=False` 描述校验而非执行。下述 P1 resolver +使 CLI/API 接受完整 manifest/目录;执行能力见 [P3 支持矩阵](querying.md#数据集全局内建支持p3) +及[定点来源](querying.md#数据集定点事件归因)。 + +```python +from pt_snap_cli.core.dataset_contract import QueryScope, validate_dataset + +inspection = validate_dataset("/capture/snapshot.pkl.msinsight") +QueryScope("event_range", device_id=0, start_event_id=1, end_event_id=3).validate( + inspection.manifest +) +assert not inspection.structured_frames +``` + +## 完整数据集 focus 与寻址(P1) + +已搬迁的兼容 v1 目录或其 `manifest.json` 可直接 focus;不要求原始 pickle、导入 +metadata、辅助引用表或扩展表: + +```bash +pt-snap focus /capture/snapshot.pkl.msinsight --device 0 --json +pt-snap overview --json +pt-snap query --template-use event --params '{"id": 3}' --json +pt-snap query --template-use event --slice 1 --json +``` + +`overview.dataset` 输出格式、manifest 版本/状态、内容指纹、真实/边界事件数、逐设备 +分片路径/范围及能力。事件范围排除负 ID 合成行(单库 overview 也如此)。支持文本调用栈, +不保证每行有栈;P0 结构化 frame 校验仍**不可用**,未知声明不会提升。跨片内建查询 +遵循 P3 矩阵;已识别有序 frame 的查询覆盖与 P0 校验能力独立。 +外部数据集级导入 metadata 为 unavailable;逐片 metadata 不足以证明整体来源。 +下述兼容导出器提供可识别的整份 `ptSnap` 证明,其完整校验后的 metadata 为 available。 + +路径优先级仍为显式 → `PT_SNAP_DB_PATH` → 最近项目 focus → legacy global config; +设备优先级仍为显式 → focused → 首个发现设备。CLI focus 只写选定的项目/全局 focus 文件, +不写产物或检测布局;API 会话 focus 不写配置: + +```python +from pathlib import Path +from pt_snap_cli import SnapshotAnalyzer + +with SnapshotAnalyzer(Path("/capture/snapshot.pkl.msinsight"), device_id=0) as analyzer: + event = analyzer.execute_query("event", {"id": 3}) + page = analyzer.execute_query("event", slice_index=1, exact_total=True) + print(event["scope"], page["total_is_exact"]) +``` + +数据集 focus 的 `event` 模板支持有界寻址:真实 ID 路由至所在片,闭区间 `min_id/max_id` +可跨片并全局筛选/排序/分页,`--slice`/`slice_index` 限制为该片真实事件。负 ID 不能作为整份数据集 +selector。输出 `scope` 标识实际 DB/device/slice、受限真实区间、数据集指纹和边界排除。 +总数和分页针对该 scope,**不是**隐式截断的整份数据集。CLI `effective_params` 的调用者 +参数默认值另受 `scope` 约束。定点 `active_blocks_at_event` 和 +`active_memory_callstack_at_event` 现于分组/排名前批量解析 alloc/free/完整栈来源, +原版无扩展产物也适用。生命周期身份、独立来源/frame 覆盖及版本化有序 frame reader +接口详见[定点归因](querying.md#数据集定点事件归因)。数据集全局内建查询遵循 P3 矩阵 +及累计工作上限;任意 SQL/自定义 override 明确失败,不静默选首片/最新片。单库真实 +事件 peak/gap 与全部 action 完整文本栈统计版本在相同合同中明确说明。 + +`core.dataset_resolver.DatasetResolver.inspect(path)` 返回不可变 `ResolvedDataset` +(单库返回 `None`);其 `paths(QueryScope(...))` 可寻址跨片范围的全部文件,但不执行查询。 +每次调用重新校验最终化 P0 合同并对 manifest/所有成员取内容哈希,不保存无界 manifest/ +连接缓存。哈希按有界块读取,工作量与产物大小成正比,不宣称性能收益。Context LRU 默认 +最多四个;数据集 generation 变化在下次 lookup 使旧 Context 失效,即使 size/mtime 相同。 +analyzer 自有缓存在退出时关闭,注入缓存仍由调用者所有;校验逐片打开并关闭连接。 + +building、缺片/矛盾片、未知基础版本、不安全路径在 focus 写入或查询前拒绝。 +**任何** SQLite 打开之前,裸 P0 validator 与 resolver 均先检查**全部**成员/父目录别名及 +存活/悬空 WAL、journal、SHM sidecar。最终化、已关闭的兼容 v1 成员使用 +`mode=ro&immutable=1`,缓存查询的 Context 连接也一样;接受固定原始生产端已 checkpoint、 +无 sidecar 的 WAL header,不创建 sidecar、不改变字节。immutable 只适用于已关闭最终化 +产物,**不是读取 live WAL 的捷径**,不保证并发 writer 安全。不修复、不 checkpoint、 +不改变 journal mode、不迁移、不删除。原生数据集仍拒绝持久 WAL,单库 Context 默认仍 +为不带 immutable 的 `mode=ro`。Cache 身份包含传输模式与 generation。这是只读校验, +不是文件系统沙箱或生产端锁。 + +native-v2 私有 staging **没有已发布 manifest**,不能作为数据集;单个 DB 仍由单库 +入口读取。已发布原生格式与兼容导出是下述独立显式协议。跨片生命周期执行遵循 P3 矩阵, +安全读取原版 WAL 不等于 GUI 验收。 + +## 已发布原生数据集(P2) + +这是 pt-snap **原创、独立命名和版本化**的协议,不是重标记的 P0 产物。ImportService/CLI +将内部原生回放结果发布至 `/<完整源文件名>.pt-snap-native-v2/`。根目录只能 +包含 `manifest.json` 与声明的 `device_` 目录;各设备目录只能有规范的 +`slice_.db` 成员。 + +原生 manifest 必需字段: + +| 字段 | 合同 | +| --- | --- | +| `format` | 严格为 `pt-snap-native-v2`,与 compatibility-v1 独立分派 | +| `schemaVersion`, `status` | 原生 manifest 版本 `1`,只允许 `complete` | +| `sourceFile` | 非空源 basename,与 `metadata.source_name` 一致,分析不打开源文件 | +| `identity` | 精确字段 `sourceSha256`、`requestedDevice`、`eventsPerSlice`、`outputFormat`、`manifestContractVersion`、`formatContractVersion`、`importContractVersion`、`extensionContractVersion`、`metadataSchemaVersion`、`importFormatVersion` | +| `metadata` | 完整已有 `ImportMetadata` 对象,与每片唯一 metadata 行一致 | +| `omittedDevices` | 不重复的未选/空/仅静态源设备位置,与所选设备不相交 | +| `devices` | 规范十进制 key,正 `eventCount/sliceCount`,全部按序 `readySlices` 及按序 `slices` | +| 各 slice | `index`、`startPosition/endPosition`、`startEventId/endEventId`、规范相对 `file`、`ready: true`、已关闭成员 `sha256` | + +所有语义合同版本初始为 `1`;原生 metadata schema 为 `1`、DB import format 为 `2`。 +`requestedDevice` 为 null(全部有事件设备)或一个非负整数;SHA256 为 64 位小写十六进制。 +CLI `metadata.importer_version` 仅提供信息,**不是**缓存失效轴。原生与后续兼容格式身份 +独立版本化,不生成原生 `cacheHash`,也不接受它替代缓存身份。 + +位置在各设备内连续覆盖 `0..eventCount-1`,末片之前全部满容量,末片至多容量个事件。 +ID 区间为原始时间顺序闭边界,可稀疏/非零;真实计数来自位置和实际行,不是 +`max(id)+1`。负 ID 合成行单独计数。各成员须有本设备的原生 v2 trace/block 实际表、 +`dictionary`、`callstack`、`pt_snap_metadata` 与最终化 `pt_snap_block_reference` v1。 +trace 列与单库 v2 一致(`callstackId`,非内联文本),保留 OOM action 8。block 身份/ +生命周期和文本引用须最终化、跨片一致,并引用原设备真实 ID。未知扩展声明/表不会 +提升结构化 frame 能力。 + +原生 resolver 校验全部成员哈希、实际 schema/dictionary、真实计数/范围、metadata 和 +最终化引用,保留 P1 打开前的别名/sidecar/持久 WAL 检查及有界借用 cache 所有权。 +无需源 pickle 即可 focus、metadata、overview 和单片 `event` 寻址。原生数据集 metadata +为 available,外部兼容数据集为 unavailable(下述可识别兼容证明除外)。原生模式不隐含 +迁移、修复、兼容导出、salted hash 或 GUI 验收;数据集全局内建查询遵循 P3 矩阵。定点 +来源解析通过上述共享查询路径可用。P0 schema/manifest +语义不变,仅兼容最终化 WAL 的只读传输修正见上文。 + +发布、force/focus 补偿见[导入指南](quickstart.md#可选导入原生分库数据集)。完整 manifest +只能与全部最终化成员同时可见,单个 ready 片不构成完整产物。 + +## 内部持续回放(P1) + +`core.sharded_replay_service.ShardedReplayService.stage(source, directory, +events_per_slice=..., device=None)` 是**内部生产边界**,不是新 CLI/API 导入命令。 +它只加载一次可信 pickle,每个所选非空设备只创建一个持续模拟器,从末片向前写入 +新的、调用者所有的**私有 staging 目录**。`omitted_devices` 披露全部未选/空设备位置。 +现有 `import`、`split`、`SnapshotAnalyzer` 与单库 v1/v2 读取保持不变。 + +返回的不可变结果标识 **native-v2** 分片,不是兼容 v1 产物:trace 使用 +`callstackId` 和片内 `callstack` 表,保留原生 OOM action 8 与 workspace 事件。 +原生真实 ID 可以稀疏或非零起点,但必须非负、唯一且按时间递增;窗口按列表位置计数, +不是 `max(id)+1`。这些原生片**不能**通过 `validate_dataset`。兼容导出、兼容 manifest、 +缓存/发布及数据集查询属于后续独立工作。不生成 `manifest.json`、`readySlices` 或增量 +ready 通知;仅所有设备完成回填及校验后的成功返回结果可交给后续发布层。 +失败保留调用者所有的私有部分文件,不将其宣称为完整数据集。 + +`SimulateDeviceSnapshot.replay_until(remaining_events)` 倒序回放到剩余指定数量的 +时间顺序列表事件。端点是**位置**而非 event ID;重复暂停/继续与完整 replay 共用 +hooks 和错误路径。真实事件行记录事件**之后**的状态(undo 之前)。undo 窗口首事件后, +活跃 block 观察和负 ID `segment_alloc`/`segment_map` 行重建首事件**之前**的状态; +边界行不计入容量或真实事件峰值。 + +设备内 registry 跨 writer 保留原始 block 对象,不用 hook 副本或仅地址作为身份。 +遇到真实 alloc 后,每次出现都最终指向原 alloc ID;未知/预存生命周期保持稳定负身份。 +所有 writer 关闭后才逐 DB 事务批量回填并校验。`freeEventId` 始终是 `free_completed`, +不是 `free_requested`;边界 `active_pending_free` 观察保留 state 0。同地址复用及不同 +stream 的同地址块不会合并。每个原生片加入 +`pt_snap_block_reference(blockId, stream, allocCallstack, freeCallstack)`,保留最终化的 +片外生命周期引用的文本调用栈来源。NULL 表示未观察到来源事件;空文本表示已观察事件 +没有 frames。旧有片内查询 JOIN 不自动消费这个内部表;它不是跨片查询引擎或结构化 +frame 扩展。 + +pickle 仍可能整体加载。分库只限制**单库真实事件数**,不限制 block 行数、总 staging +字节、registry 大小或峰值 RSS。合成回归及已审 expandable/多设备 fixture 覆盖回放 +等价和资源失败路径;不宣称真实上游 GUI 或性能验收。 + +## 显式 msinsight 兼容导出(P2) + +`pt-snap import snapshot.pkl --format msinsight --json`(别名 +`--format compatibility-v1`)**仅**对齐 +`Ascend/msinsight@101f65b877a267ffd5f66ea3834706057ba243e5`。支持入口是在该目标版本 +打开**同一原始 pickle**及邻接 `<原始路径>.msinsight/`,不是 GUI 直接导入独立目录。 +`--output-dir` 为 pt-snap 选择其他父目录,搬迁本身不创建上游入口。`sourceFile` 记录 +绝对原始源路径。容量默认 `500000`,可用 `--events-per-slice` 调整;默认单库及显式 +native-v2 输出仍独立且不变。 + +第一方转换器消费已关闭/最终化的原生暂存,物化精确位置内联 v1 **实际表**,去除 +未隔离原生 `callstack`,写入物理格式 metadata `1`。保留原始真实 ID、action、stream/ +指标、设备内稳定 block 身份、`free_completed` 和片外生命周期文本来源引用,不插入 +外片真实事件掩盖片内 JOIN 缺口。OOM/未知 action、非零/稀疏 ID、空/仅静态选择失败; +原生保留更广合同。所有选中成员最终化、关闭、校验后才发布 complete。 +`pt_snap_block_reference` v1 是隔离文本扩展,不是有序 frames 或跨来源查询承诺; +跨来源查询/全局聚合遵循 P3 查询矩阵。文本不能恢复原始有序 frame 数组;原版 workspace +raw-frame 丢失与字符串 OOM action 强制转换属于已知局限,不应仿制为兼容能力。 + +`cacheHash = SHA256(b"mem_snapshot_parser_v2" + 原始_pickle_字节)`,salt **前置**。 +独立 `ptSnap.identity` 字段为 `sourceSha256`、`requestedDevice`、`eventsPerSlice`、 +`outputFormat=compatibility-v1`、`manifestContractVersion=1`、`exportContractVersion=1`、 +`referenceContractVersion=1`、`metadataSchemaVersion=1`、`importFormatVersion=1`、完整 +`targetRevision`。`ptSnap` 另含 `metadata`、`omittedDevices` 和 `members`(规范相对路径 +→ 最终化 SHA256)。整体已校验所有权/身份要求每个成员/哈希及逐片 metadata 相同; +不接管未知/外部缓存或无关文件。CLI 版本变化本身不使缓存失效。 + +兼容发布 **force 也 no-replace**。已识别、身份相同的 pt-snap 缓存不加载 pickle 即复用, +force 不绕过该规则。源路径/内容、选项/合同变化、成员损坏或 GUI 修改 DB 后须换新 +`--output-dir`,不修复、不覆盖、不删除旧产物。排他发布与 focus 补偿使用已有受检事务, +发布/复用前再次核对源 SHA256;原生 force 替换与恢复语义不变。 + +内联文本成本按每个事件的 UTF-8 调用栈字节求和,不按不同栈各存一份;原生去重仍可用。 +转换暂时并存两种布局,可能留下 SQLite free pages;容量不是 RSS/registry/block/磁盘 +上限。不生成可选 frame 增强。 + +**GUI 待验收/未运行。** 固定源码 parser 在 UP_TO_DATE 前检查 complete manifest、 +非空相同 hash、每个 ready/可解析/存在成员、只读 DB/设备校验,以及 allocation cache +存在**或可构建**。派生 allocation cache 可在不运行生产者时修改 DB,mtime 或新派生表 +不能单独证明重跑。后续 GUI 验收须保留 UP_TO_DATE/无 script 调用日志并对照曲线、block、 +跨片展示。源码检查及原版产物的生产 reader 读取不是 GUI、C++ server/编译或完整差分/ +性能验收。 + +不可变固定 blob 证据: +[parser 判据](https://api.github.com/repos/Ascend/msinsight/git/blobs/40046e6a92ff5af776ee55bd4cdd4586745af3c2)、 +[salt 顺序](https://api.github.com/repos/Ascend/msinsight/git/blobs/ca421c7b33f3a0050df9e7a01e666f558801bc2f)、 +[allocation cache 判据/构建](https://api.github.com/repos/Ascend/msinsight/git/blobs/3d1ecc0ebd0d8e3eb0c984eda6b750ec4ce0e4f1)。 +Ascend 源码为 Mulan PSL v2,docs 单独为 CC BY 4.0;此最小转换器是本地原创接口事实实现, +未复制 vendor 实现,也不把 Ascend blob 自动重新许可为 MIT。已有 pt-snap 运行时 lineage +属于独立许可证/来源历史。 + +## 可重复验收与实测限制 + +[验收与有界性能](interop-acceptance.md) 提供确定性 CI、hash-first 的已有原版消费者/ +正向物理对照、同源三模式本地基线及明确待验收 GUI 清单。证据层彼此独立;负 token +仅在数据集内标识对象,不承诺跨生产端对象身份,也不允许重写真实 ID。 + +## 来源证据与限制 + +接口事实已从固定版本真实源码读取: +[manifest 校验](https://github.com/Ascend/msinsight/blob/101f65b877a267ffd5f66ea3834706057ba243e5/server/src/modules/memsnapshot/service/MemSnapshotSliceService.cpp)、 +[SQLite 按位置读取](https://github.com/Ascend/msinsight/blob/101f65b877a267ffd5f66ea3834706057ba243e5/server/src/modules/memsnapshot/database/MemSnapshotDatabase.cpp)、 +[基础 schema](https://api.github.com/repos/Ascend/msinsight/git/blobs/1fe50a5136f85365db55f4632d58f3cf7090431e)、 +[发布/回填](https://api.github.com/repos/Ascend/msinsight/git/blobs/95612509ef990ac6b7034919dae4938cc6acf3ad)。 +这些源码包含 Huawei Mulan PSL v2 声明。本文合同、Python 校验器和合成测试是本地原创的 +接口事实实现,没有向包内复制上游实现;上游 docs 单独使用 CC BY 4.0,不能替代源码声明。 + +complete 就绪一致性和规范路径规则刻意严于上游 parser(上游也支持 building)。 +合成合同测试/真实源函数对照不是 GUI 验收、原生导出器验收、性能测量,也不证明其他版本互通。 diff --git a/skills/pt-snap-helper/SKILL.md b/skills/pt-snap-helper/SKILL.md index 1468fd6..0fc07a8 100644 --- a/skills/pt-snap-helper/SKILL.md +++ b/skills/pt-snap-helper/SKILL.md @@ -181,6 +181,35 @@ Missing-focus recovery below is the explicit metadata-first exception: after user confirmation, inspect metadata first, then obtain/reuse capabilities and overview and apply these same checks. Metadata alone does not replace overview. +### Complete dataset handoff + +Accept only an explicit standalone DB or complete validated compatibility-v1 / +pt-snap-native-v2 dataset directory or `manifest.json`, not arbitrary or sibling +directories. Existence is not validation. Preserve the chosen target/device/scope +without import or persisted focus. Validate `overview.dataset.status`, `format`, +`manifest_path`, `fingerprint`, real bounds and device; reuse requires unchanged +fingerprint. Valid external original artifacts can report `unavailable` / +`metadata_missing` with unknown provenance; invalid metadata, incomplete validation +and every other unknown reason still stop. + +Tell the diagnostic skill to check each `dataset_support.supported` catalog entry; +all eleven built-ins have declared semantics, custom SQL/overrides do not. Keep +source/range coverage separate from `has_more`, `truncated` and exact totals; +point queries use `scope.source_coverage.range_complete`, range queries use +`scope.range_complete`. Pass `allocation_source_complete` / `ordered_frames_complete` +when available, not an inferred complete claim. Use `lifecycle_id` within unchanged +fingerprint/device with `allocation_source` / `free_source` and original alloc/free +IDs, never address/local stack ID. +`stack_id` / `source_stack_id` is canonical text/raw-frame identity; local IDs, +source event/slice and `text_kind` are provenance. Recognized ptSnapOrderedFrames +version 1 requires actual schema/per-event coverage; otherwise downgrade to +text-only, never reconstruct frames. `callstack_analysis` legacy `alloc_count` +means ALL real stack-bearing actions, not allocation-only, and size activity is +not live bytes. Preserve included-byte denominators and SAME selected-metric-event +report gap/coverage; one query/report cumulative 100000 fetched/output rows /64 MiB +serialized values is NOT an RSS ceiling. Budget failure is no partial global proof. +This helper still only hands off; it does not execute analysis queries. + ## Routing matrix Choose one next skill. If the goal is still ambiguous after this table, ask @@ -212,8 +241,11 @@ only has those, stop and ask for a `.db` or a trusted pickle. ### Existing SnapshotDB -A SnapshotDB is a pt-snap SQLite database (typically `.db`) with a -`dictionary` table. When the user already has one: +A standalone SnapshotDB is a SQLite database (typically `.db`) with a +`dictionary` table. An explicitly selected complete validated compatibility-v1 or +pt-snap-native-v2 dataset directory or `manifest.json` is also an analysis target. +Arbitrary directories and single ready shards are not complete datasets. +When the user already has one: 1. Confirm the goal is leak, peak, or fragmentation. 2. Tell the next skill to start with `pt-snap capabilities --json` and diff --git a/skills/pt-snap-memory-fragmentation/SKILL.md b/skills/pt-snap-memory-fragmentation/SKILL.md index 0aba201..488cf5c 100644 --- a/skills/pt-snap-memory-fragmentation/SKILL.md +++ b/skills/pt-snap-memory-fragmentation/SKILL.md @@ -26,8 +26,9 @@ derived from this workflow. Optional inputs: - An event range to investigate, expressed as the `` and - `` event IDs. Use `0` as `` unless a validated - explicit lower bound applies. Omit every upper-bound parameter while the + `` event IDs. Use the validated selected device's `first_event_id` + as `` (zero only if its actual first real ID is zero). Native IDs + can be sparse/nonzero; never remap them. Omit every upper-bound parameter while the scope has no validated upper bound. - A positive page size for timeline and event pagination; default `1000`. - Selected peak or pressure event IDs for active-block attribution. @@ -64,7 +65,8 @@ to select it and stop until it is explicit. Do not run `pt-snap focus " --device --template-use active_memory_call The range selects the attribution event, not the allocation dates of live blocks. Keep blocks allocated before `` that are still live at that event; do not add an allocation-window filter. Record the actual attribution event ID -alongside the range peak and verify they match. Do not use `report peak-memory` -for this branch: it selects its own unfiltered full-trace peak. +alongside the range peak and verify they match. Keep the explicit-event query +workflow in this branch. An unbounded `report peak-memory` selects an unfiltered +full-trace peak; the report also supports `--start-id` / `--end-id`, but a bounded +report must use the SAME verified bounds and event, never substitute another peak. #### Separate full-trace comparison (only when explicitly requested) diff --git a/skills/pt-snap-memory-leak/SKILL.md b/skills/pt-snap-memory-leak/SKILL.md index 0ef7176..0e25502 100644 --- a/skills/pt-snap-memory-leak/SKILL.md +++ b/skills/pt-snap-memory-leak/SKILL.md @@ -50,11 +50,11 @@ If the user did not provide a database or device, inspect the current state: pt-snap focus ``` -Use the displayed database and focused device only when both are present, the database file exists, and the user has not requested another target. If the database or device remains ambiguous, ask the user to select it before analysis. +Use the displayed database and focused device only when both are present, the selected target exists and passes the overview validation below, and the user has not requested another target. If the database or device remains ambiguous, ask the user to select it before analysis. Do not run `pt-snap focus ` or otherwise persist focus from this skill. Pass the database and device explicitly to every diagnostic query. -This skill accepts SnapshotDB files only. Do not run `pt-snap import` or deserialize a pickle snapshot. If the user has only a pickle file, stop and explain that importing requires a separate, explicit trusted-input decision because pickle loading is not a sandbox. +This skill accepts standalone SnapshotDB files or an explicitly selected complete validated dataset directory or `manifest.json` only. Do not run `pt-snap import` or deserialize a pickle snapshot. If the user has only a pickle file, stop and explain that importing requires a separate, explicit trusted-input decision because pickle loading is not a sandbox. #### Missing focus target with sibling candidates @@ -100,6 +100,48 @@ target's overview or device. Confirm these templates exist in the capabilities catalog before diagnosis: `memory_peak`, `allocator_gap`, `event`, `block`, `leak_detection`, `active_memory_callstack_at_event`, `active_blocks_at_event`, `preexisting_live`, `freed_block_lifetime`. If the catalog or overview fails, stop and report the exact failure. Do not silently substitute raw SQL for a missing core template. +## Validated complete dataset scope + +Accept only the user's explicit standalone file or complete validated +compatibility-v1 / pt-snap-native-v2 dataset directory or `manifest.json`. +Existence alone is not validation. Never infer arbitrary or sibling directories, +auto-select a device, import pickle, or persist focus. Keep the same explicit +target/device pair in every command and record full-device versus selected range. +For a dataset, validate `overview.dataset.status`, `format`, `manifest_path`, +`fingerprint`, real bounds and selected device. Reuse requires the same fingerprint. +Read `overview.import_metadata`: a valid external original artifact may report +`unavailable` / `metadata_missing`; record unknown provenance. This is not permission +to allow invalid metadata, incomplete validation, or any other unknown reason. + +Check every chosen catalog entry's `dataset_support.supported`. All eleven built-ins +have declared dataset semantics; arbitrary SQL and custom built-in-name overrides +are unsupported. Never concatenate shard-local top-N or sum different device peaks. +Use `lifecycle_id` within the same fingerprint/device plus original `allocEventId`, +`allocation_source` and `free_source`; completed frees have action 6, not pending +free_requested. Address and shard-local stack IDs are not lifecycle identity. +Stored `state_scope` is an observation, not reconstructed state at arbitrary E. +Dataset `stack_id` / `source_stack_id` uses full canonical text or covered ordered +raw frames; `local_stack_id`, source event/slice and `text_kind` are provenance. +Never merge shortened labels. Keep captured literal missing-label text separate +from missing references. `callstack_analysis` semantics_version=2 counts ALL real +stack-bearing actions in legacy `alloc_count`, not allocation-only; `total_size` +is activity, not live bytes, and non-NULL empty text activity remains its own group. + +Check `scope.range_complete` for ranges and +`scope.source_coverage.range_complete` for point queries; source/frame coverage is +separate from `has_more`, `truncated`, and exact totals. Retain +`allocation_source_complete` and `ordered_frames_complete`. Only recognized +ptSnapOrderedFrames version 1 with actual schema and per-event coverage supplies +ordered frames (including duplicates); missing/unknown/uncovered means text-only. +Never reconstruct frames or claim hierarchical evidence from text alone. +One query/report shares diminishing time and cumulative 100000 fetched/output rows +and 64 MiB serialized-value budgets, NOT an RSS ceiling. Budget failure returns no +partial global proof; narrow scope or ask for an explicit member, never raw SQL. +Percentages use included group bytes (standalone SQL or dataset post-merge) after filtering/top-N and before the caller's +row cap, not the whole active counter; report coverage uses active at the SAME +selected metric event. Leak candidates require terminal full-device dataset scope, +not an arbitrary `--slice`; end-live survival is not a confirmed leak. + ## Optional compact stack evidence For long stack text, check the capabilities catalog for `stack_bytes` on `event` @@ -189,7 +231,7 @@ Use the final event ID from Step 1: pt-snap query '' --device --template-use active_memory_callstack_at_event --params '{"event_id":,"include_static":true,"min_size":0,"top_n":20}' ``` -This template has no `offset`, and `-n` cannot raise the CTE `top_n` cap. If +This template has no `offset`, and `-n` cannot raise the template `top_n` cap. If `has_more` or `truncated` is true, increase `top_n` instead of concluding from the first ranked page. diff --git a/skills/pt-snap-memory-peak-breakdown/SKILL.md b/skills/pt-snap-memory-peak-breakdown/SKILL.md index 8267c8b..1d7a275 100644 --- a/skills/pt-snap-memory-peak-breakdown/SKILL.md +++ b/skills/pt-snap-memory-peak-breakdown/SKILL.md @@ -11,7 +11,8 @@ Explain what memory was live at selected active, allocated, or reserved high-wat events. This is a point-in-time SnapshotDB analysis, not an end-of-trace leak diagnosis. It must not claim a fragmentation or OOM root cause. -Use only an existing pt-snap SnapshotDB (`.db`). Never open, import, inspect, or +Use only an existing standalone SnapshotDB (`.db`) or an explicitly selected, +complete validated dataset directory or `manifest.json`. Never open, import, inspect, or deserialize pickle (`.pkl` or `.pickle`) input. ## Required Inputs @@ -169,6 +170,45 @@ Stop and report the exact failure if it fails or the options used below are absent. A complete successful help result from the same CLI/Python environment and version may be reused. +## Validated complete dataset scope + +Accept only an explicit standalone file or complete validated compatibility-v1 / +pt-snap-native-v2 dataset directory or `manifest.json`, never arbitrary or sibling +directories. Existence alone is not validation. Keep explicit target/device/scope; +do not auto-select devices, import pickle or persist focus. Validate +`overview.dataset.status`, `format`, `manifest_path`, `fingerprint`, real bounds +and device. Reuse requires unchanged fingerprint and import provenance. +Valid original no-extension artifacts may have `unavailable` / `metadata_missing`; +record unknown provenance, but stop on invalid metadata, incomplete validation or +any other unknown reason. Require each catalog entry's `dataset_support.supported`. +All eleven built-ins declare dataset semantics; custom SQL and built-in-name +overrides are unsupported. Never concatenate local top-N or sum device peaks. + +Use declared `lifecycle_id`, `allocation_source` and `free_source` plus original +alloc/free IDs within the same fingerprint/device; completed free is action 6, +not pending free_requested. Address or local stack ID is not lifecycle identity. +`state_scope` is a stored observation, not state reconstructed at arbitrary E. +Dataset `stack_id` / `source_stack_id` groups canonical full text or covered ordered +raw frames; `local_stack_id`, event/slice and `text_kind` are provenance. Do not +merge shortened text or captured literal missing-label strings with missing links. +`callstack_analysis` semantics_version=2 legacy `alloc_count` counts ALL real +stack-bearing actions, not allocation-only; `total_size` is activity, not live +bytes. Non-NULL empty text activity remains separate from a missing reference. + +Read `scope.range_complete` for ranges and +`scope.source_coverage.range_complete` for point queries. Source/frame coverage, +`allocation_source_complete`, and `ordered_frames_complete` are separate from +`has_more`, `truncated` and exact totals. Only recognized ptSnapOrderedFrames +version 1 with real schema and per-event coverage supplies ordered frames, +retaining order/duplicates; missing/unknown/uncovered means text-only. Never +reconstruct frames or claim hierarchical evidence from formatted text. +One query/report shares diminishing time and cumulative 100000 fetched/output rows +and 64 MiB serialized values, NOT an RSS ceiling. Failure gives no partial global +proof; narrow scope or ask for an explicit member, never substitute raw SQL. +Percentages use included group bytes (standalone SQL or dataset post-merge) after filtering/top-N, before the caller row cap, +not the whole active counter. Report gap/coverage uses active at the SAME selected +metric event, not an independently occurring active peak. + ## Optional compact stack evidence When long stacks dominate output bytes, first confirm `stack_bytes` in the @@ -191,7 +231,9 @@ attribution once for multiple metrics; retain each metric's counters and gaps. ## Full-Trace Workflow -`pt-snap report peak-memory` is full-trace only. Run it once for each metric so +`pt-snap report peak-memory` without range bounds selects the full real trace. +It also supports `--start-id` / `--end-id`; never mix bounded and unbounded results. +Run the unbounded report once for each metric so each metric is attributed at its own peak event: ```bash @@ -252,9 +294,12 @@ equals `returned` unless `--exact-total` is set. ## Event-Range Workflow -Do not use `report peak-memory` for a bounded range because the report accepts -no `start_id` or `end_id` and is full-trace only. Run the range-capable templates -with the same inclusive bounds instead: +The report supports `--start-id` / `--end-id` with the same inclusive real-ID +bounds on standalone files and complete datasets. This workflow keeps the +range-capable templates followed by explicit-event attribution below, preserving +its empty/NULL/range guards. An optional bounded report must use the SAME validated +bounds and selected metric; never substitute an unbounded report or another event. +Run the templates with the same inclusive bounds: ```bash pt-snap query "" --device --template-use memory_peak --params '{"start_id": , "end_id": }' @@ -275,7 +320,7 @@ pt-snap query "" --device --template-use active_blocks_at_event --p ``` `active_memory_callstack_at_event` has no `offset`, and `-n` cannot raise the -CTE `top_n` cap. Check `has_more`, `truncated`, and `total_is_exact` after every +template `top_n` cap. Check `has_more`, `truncated`, and `total_is_exact` after every query. If completeness is unconfirmed, first ensure `-n` is at least `top_n + 2` with static inclusion, then increase `top_n` and recompute `` when full dynamic coverage is needed. A full ranking window sets `has_more=true` diff --git a/src/pt_snap_cli/api.py b/src/pt_snap_cli/api.py index 6ca2d90..1e462ef 100644 --- a/src/pt_snap_cli/api.py +++ b/src/pt_snap_cli/api.py @@ -32,7 +32,11 @@ class TemplateSummaryPayload(TypedDict): category: str | None -class QueryResultPayload(TypedDict): +class _QueryScopePayload(TypedDict, total=False): + scope: dict[str, object] + + +class QueryResultPayload(_QueryScopePayload): total: int returned: int device_id: int | None @@ -209,6 +213,7 @@ def execute_query( *, exact_total: bool = False, timeout_s: float | None = None, + slice_index: int | None = None, ) -> QueryResultPayload: self._ensure_open() try: @@ -220,10 +225,11 @@ def execute_query( max_rows=max_rows, exact_total=exact_total, timeout_s=timeout_s, + slice_index=slice_index, ) except FocusNotConfiguredError as exc: raise RuntimeError("No database configured. Call set_focus() first.") from exc - return { + payload: QueryResultPayload = { "total": result.total, "returned": result.returned, "device_id": result.device_id, @@ -235,6 +241,9 @@ def execute_query( "total_is_exact": result.total_is_exact, "timeout_s": result.timeout_s, } + if result.scope is not None: + payload["scope"] = result.scope + return payload def list_capabilities(self) -> dict[str, object]: self._ensure_open() diff --git a/src/pt_snap_cli/bundled_skills/pt-snap-helper/SKILL.md b/src/pt_snap_cli/bundled_skills/pt-snap-helper/SKILL.md index 1468fd6..0fc07a8 100644 --- a/src/pt_snap_cli/bundled_skills/pt-snap-helper/SKILL.md +++ b/src/pt_snap_cli/bundled_skills/pt-snap-helper/SKILL.md @@ -181,6 +181,35 @@ Missing-focus recovery below is the explicit metadata-first exception: after user confirmation, inspect metadata first, then obtain/reuse capabilities and overview and apply these same checks. Metadata alone does not replace overview. +### Complete dataset handoff + +Accept only an explicit standalone DB or complete validated compatibility-v1 / +pt-snap-native-v2 dataset directory or `manifest.json`, not arbitrary or sibling +directories. Existence is not validation. Preserve the chosen target/device/scope +without import or persisted focus. Validate `overview.dataset.status`, `format`, +`manifest_path`, `fingerprint`, real bounds and device; reuse requires unchanged +fingerprint. Valid external original artifacts can report `unavailable` / +`metadata_missing` with unknown provenance; invalid metadata, incomplete validation +and every other unknown reason still stop. + +Tell the diagnostic skill to check each `dataset_support.supported` catalog entry; +all eleven built-ins have declared semantics, custom SQL/overrides do not. Keep +source/range coverage separate from `has_more`, `truncated` and exact totals; +point queries use `scope.source_coverage.range_complete`, range queries use +`scope.range_complete`. Pass `allocation_source_complete` / `ordered_frames_complete` +when available, not an inferred complete claim. Use `lifecycle_id` within unchanged +fingerprint/device with `allocation_source` / `free_source` and original alloc/free +IDs, never address/local stack ID. +`stack_id` / `source_stack_id` is canonical text/raw-frame identity; local IDs, +source event/slice and `text_kind` are provenance. Recognized ptSnapOrderedFrames +version 1 requires actual schema/per-event coverage; otherwise downgrade to +text-only, never reconstruct frames. `callstack_analysis` legacy `alloc_count` +means ALL real stack-bearing actions, not allocation-only, and size activity is +not live bytes. Preserve included-byte denominators and SAME selected-metric-event +report gap/coverage; one query/report cumulative 100000 fetched/output rows /64 MiB +serialized values is NOT an RSS ceiling. Budget failure is no partial global proof. +This helper still only hands off; it does not execute analysis queries. + ## Routing matrix Choose one next skill. If the goal is still ambiguous after this table, ask @@ -212,8 +241,11 @@ only has those, stop and ask for a `.db` or a trusted pickle. ### Existing SnapshotDB -A SnapshotDB is a pt-snap SQLite database (typically `.db`) with a -`dictionary` table. When the user already has one: +A standalone SnapshotDB is a SQLite database (typically `.db`) with a +`dictionary` table. An explicitly selected complete validated compatibility-v1 or +pt-snap-native-v2 dataset directory or `manifest.json` is also an analysis target. +Arbitrary directories and single ready shards are not complete datasets. +When the user already has one: 1. Confirm the goal is leak, peak, or fragmentation. 2. Tell the next skill to start with `pt-snap capabilities --json` and diff --git a/src/pt_snap_cli/bundled_skills/pt-snap-memory-fragmentation/SKILL.md b/src/pt_snap_cli/bundled_skills/pt-snap-memory-fragmentation/SKILL.md index 0aba201..488cf5c 100644 --- a/src/pt_snap_cli/bundled_skills/pt-snap-memory-fragmentation/SKILL.md +++ b/src/pt_snap_cli/bundled_skills/pt-snap-memory-fragmentation/SKILL.md @@ -26,8 +26,9 @@ derived from this workflow. Optional inputs: - An event range to investigate, expressed as the `` and - `` event IDs. Use `0` as `` unless a validated - explicit lower bound applies. Omit every upper-bound parameter while the + `` event IDs. Use the validated selected device's `first_event_id` + as `` (zero only if its actual first real ID is zero). Native IDs + can be sparse/nonzero; never remap them. Omit every upper-bound parameter while the scope has no validated upper bound. - A positive page size for timeline and event pagination; default `1000`. - Selected peak or pressure event IDs for active-block attribution. @@ -64,7 +65,8 @@ to select it and stop until it is explicit. Do not run `pt-snap focus " --device --template-use active_memory_call The range selects the attribution event, not the allocation dates of live blocks. Keep blocks allocated before `` that are still live at that event; do not add an allocation-window filter. Record the actual attribution event ID -alongside the range peak and verify they match. Do not use `report peak-memory` -for this branch: it selects its own unfiltered full-trace peak. +alongside the range peak and verify they match. Keep the explicit-event query +workflow in this branch. An unbounded `report peak-memory` selects an unfiltered +full-trace peak; the report also supports `--start-id` / `--end-id`, but a bounded +report must use the SAME verified bounds and event, never substitute another peak. #### Separate full-trace comparison (only when explicitly requested) diff --git a/src/pt_snap_cli/bundled_skills/pt-snap-memory-leak/SKILL.md b/src/pt_snap_cli/bundled_skills/pt-snap-memory-leak/SKILL.md index 0ef7176..0e25502 100644 --- a/src/pt_snap_cli/bundled_skills/pt-snap-memory-leak/SKILL.md +++ b/src/pt_snap_cli/bundled_skills/pt-snap-memory-leak/SKILL.md @@ -50,11 +50,11 @@ If the user did not provide a database or device, inspect the current state: pt-snap focus ``` -Use the displayed database and focused device only when both are present, the database file exists, and the user has not requested another target. If the database or device remains ambiguous, ask the user to select it before analysis. +Use the displayed database and focused device only when both are present, the selected target exists and passes the overview validation below, and the user has not requested another target. If the database or device remains ambiguous, ask the user to select it before analysis. Do not run `pt-snap focus ` or otherwise persist focus from this skill. Pass the database and device explicitly to every diagnostic query. -This skill accepts SnapshotDB files only. Do not run `pt-snap import` or deserialize a pickle snapshot. If the user has only a pickle file, stop and explain that importing requires a separate, explicit trusted-input decision because pickle loading is not a sandbox. +This skill accepts standalone SnapshotDB files or an explicitly selected complete validated dataset directory or `manifest.json` only. Do not run `pt-snap import` or deserialize a pickle snapshot. If the user has only a pickle file, stop and explain that importing requires a separate, explicit trusted-input decision because pickle loading is not a sandbox. #### Missing focus target with sibling candidates @@ -100,6 +100,48 @@ target's overview or device. Confirm these templates exist in the capabilities catalog before diagnosis: `memory_peak`, `allocator_gap`, `event`, `block`, `leak_detection`, `active_memory_callstack_at_event`, `active_blocks_at_event`, `preexisting_live`, `freed_block_lifetime`. If the catalog or overview fails, stop and report the exact failure. Do not silently substitute raw SQL for a missing core template. +## Validated complete dataset scope + +Accept only the user's explicit standalone file or complete validated +compatibility-v1 / pt-snap-native-v2 dataset directory or `manifest.json`. +Existence alone is not validation. Never infer arbitrary or sibling directories, +auto-select a device, import pickle, or persist focus. Keep the same explicit +target/device pair in every command and record full-device versus selected range. +For a dataset, validate `overview.dataset.status`, `format`, `manifest_path`, +`fingerprint`, real bounds and selected device. Reuse requires the same fingerprint. +Read `overview.import_metadata`: a valid external original artifact may report +`unavailable` / `metadata_missing`; record unknown provenance. This is not permission +to allow invalid metadata, incomplete validation, or any other unknown reason. + +Check every chosen catalog entry's `dataset_support.supported`. All eleven built-ins +have declared dataset semantics; arbitrary SQL and custom built-in-name overrides +are unsupported. Never concatenate shard-local top-N or sum different device peaks. +Use `lifecycle_id` within the same fingerprint/device plus original `allocEventId`, +`allocation_source` and `free_source`; completed frees have action 6, not pending +free_requested. Address and shard-local stack IDs are not lifecycle identity. +Stored `state_scope` is an observation, not reconstructed state at arbitrary E. +Dataset `stack_id` / `source_stack_id` uses full canonical text or covered ordered +raw frames; `local_stack_id`, source event/slice and `text_kind` are provenance. +Never merge shortened labels. Keep captured literal missing-label text separate +from missing references. `callstack_analysis` semantics_version=2 counts ALL real +stack-bearing actions in legacy `alloc_count`, not allocation-only; `total_size` +is activity, not live bytes, and non-NULL empty text activity remains its own group. + +Check `scope.range_complete` for ranges and +`scope.source_coverage.range_complete` for point queries; source/frame coverage is +separate from `has_more`, `truncated`, and exact totals. Retain +`allocation_source_complete` and `ordered_frames_complete`. Only recognized +ptSnapOrderedFrames version 1 with actual schema and per-event coverage supplies +ordered frames (including duplicates); missing/unknown/uncovered means text-only. +Never reconstruct frames or claim hierarchical evidence from text alone. +One query/report shares diminishing time and cumulative 100000 fetched/output rows +and 64 MiB serialized-value budgets, NOT an RSS ceiling. Budget failure returns no +partial global proof; narrow scope or ask for an explicit member, never raw SQL. +Percentages use included group bytes (standalone SQL or dataset post-merge) after filtering/top-N and before the caller's +row cap, not the whole active counter; report coverage uses active at the SAME +selected metric event. Leak candidates require terminal full-device dataset scope, +not an arbitrary `--slice`; end-live survival is not a confirmed leak. + ## Optional compact stack evidence For long stack text, check the capabilities catalog for `stack_bytes` on `event` @@ -189,7 +231,7 @@ Use the final event ID from Step 1: pt-snap query '' --device --template-use active_memory_callstack_at_event --params '{"event_id":,"include_static":true,"min_size":0,"top_n":20}' ``` -This template has no `offset`, and `-n` cannot raise the CTE `top_n` cap. If +This template has no `offset`, and `-n` cannot raise the template `top_n` cap. If `has_more` or `truncated` is true, increase `top_n` instead of concluding from the first ranked page. diff --git a/src/pt_snap_cli/bundled_skills/pt-snap-memory-peak-breakdown/SKILL.md b/src/pt_snap_cli/bundled_skills/pt-snap-memory-peak-breakdown/SKILL.md index 8267c8b..1d7a275 100644 --- a/src/pt_snap_cli/bundled_skills/pt-snap-memory-peak-breakdown/SKILL.md +++ b/src/pt_snap_cli/bundled_skills/pt-snap-memory-peak-breakdown/SKILL.md @@ -11,7 +11,8 @@ Explain what memory was live at selected active, allocated, or reserved high-wat events. This is a point-in-time SnapshotDB analysis, not an end-of-trace leak diagnosis. It must not claim a fragmentation or OOM root cause. -Use only an existing pt-snap SnapshotDB (`.db`). Never open, import, inspect, or +Use only an existing standalone SnapshotDB (`.db`) or an explicitly selected, +complete validated dataset directory or `manifest.json`. Never open, import, inspect, or deserialize pickle (`.pkl` or `.pickle`) input. ## Required Inputs @@ -169,6 +170,45 @@ Stop and report the exact failure if it fails or the options used below are absent. A complete successful help result from the same CLI/Python environment and version may be reused. +## Validated complete dataset scope + +Accept only an explicit standalone file or complete validated compatibility-v1 / +pt-snap-native-v2 dataset directory or `manifest.json`, never arbitrary or sibling +directories. Existence alone is not validation. Keep explicit target/device/scope; +do not auto-select devices, import pickle or persist focus. Validate +`overview.dataset.status`, `format`, `manifest_path`, `fingerprint`, real bounds +and device. Reuse requires unchanged fingerprint and import provenance. +Valid original no-extension artifacts may have `unavailable` / `metadata_missing`; +record unknown provenance, but stop on invalid metadata, incomplete validation or +any other unknown reason. Require each catalog entry's `dataset_support.supported`. +All eleven built-ins declare dataset semantics; custom SQL and built-in-name +overrides are unsupported. Never concatenate local top-N or sum device peaks. + +Use declared `lifecycle_id`, `allocation_source` and `free_source` plus original +alloc/free IDs within the same fingerprint/device; completed free is action 6, +not pending free_requested. Address or local stack ID is not lifecycle identity. +`state_scope` is a stored observation, not state reconstructed at arbitrary E. +Dataset `stack_id` / `source_stack_id` groups canonical full text or covered ordered +raw frames; `local_stack_id`, event/slice and `text_kind` are provenance. Do not +merge shortened text or captured literal missing-label strings with missing links. +`callstack_analysis` semantics_version=2 legacy `alloc_count` counts ALL real +stack-bearing actions, not allocation-only; `total_size` is activity, not live +bytes. Non-NULL empty text activity remains separate from a missing reference. + +Read `scope.range_complete` for ranges and +`scope.source_coverage.range_complete` for point queries. Source/frame coverage, +`allocation_source_complete`, and `ordered_frames_complete` are separate from +`has_more`, `truncated` and exact totals. Only recognized ptSnapOrderedFrames +version 1 with real schema and per-event coverage supplies ordered frames, +retaining order/duplicates; missing/unknown/uncovered means text-only. Never +reconstruct frames or claim hierarchical evidence from formatted text. +One query/report shares diminishing time and cumulative 100000 fetched/output rows +and 64 MiB serialized values, NOT an RSS ceiling. Failure gives no partial global +proof; narrow scope or ask for an explicit member, never substitute raw SQL. +Percentages use included group bytes (standalone SQL or dataset post-merge) after filtering/top-N, before the caller row cap, +not the whole active counter. Report gap/coverage uses active at the SAME selected +metric event, not an independently occurring active peak. + ## Optional compact stack evidence When long stacks dominate output bytes, first confirm `stack_bytes` in the @@ -191,7 +231,9 @@ attribution once for multiple metrics; retain each metric's counters and gaps. ## Full-Trace Workflow -`pt-snap report peak-memory` is full-trace only. Run it once for each metric so +`pt-snap report peak-memory` without range bounds selects the full real trace. +It also supports `--start-id` / `--end-id`; never mix bounded and unbounded results. +Run the unbounded report once for each metric so each metric is attributed at its own peak event: ```bash @@ -252,9 +294,12 @@ equals `returned` unless `--exact-total` is set. ## Event-Range Workflow -Do not use `report peak-memory` for a bounded range because the report accepts -no `start_id` or `end_id` and is full-trace only. Run the range-capable templates -with the same inclusive bounds instead: +The report supports `--start-id` / `--end-id` with the same inclusive real-ID +bounds on standalone files and complete datasets. This workflow keeps the +range-capable templates followed by explicit-event attribution below, preserving +its empty/NULL/range guards. An optional bounded report must use the SAME validated +bounds and selected metric; never substitute an unbounded report or another event. +Run the templates with the same inclusive bounds: ```bash pt-snap query "" --device --template-use memory_peak --params '{"start_id": , "end_id": }' @@ -275,7 +320,7 @@ pt-snap query "" --device --template-use active_blocks_at_event --p ``` `active_memory_callstack_at_event` has no `offset`, and `-n` cannot raise the -CTE `top_n` cap. Check `has_more`, `truncated`, and `total_is_exact` after every +template `top_n` cap. Check `has_more`, `truncated`, and `total_is_exact` after every query. If completeness is unconfirmed, first ensure `-n` is at least `top_n + 2` with static inclusion, then increase `top_n` and recompute `` when full dynamic coverage is needed. A full ranking window sets `has_more=true` diff --git a/src/pt_snap_cli/cli.py b/src/pt_snap_cli/cli.py index 620f864..cfa20e1 100644 --- a/src/pt_snap_cli/cli.py +++ b/src/pt_snap_cli/cli.py @@ -94,6 +94,7 @@ json_success, ) from pt_snap_cli.core.error_codes import DATABASE_NOT_FOUND, ERROR, INVALID_PARAMETER +from pt_snap_cli.core.errors import InvalidParameterError from pt_snap_cli.query.config import OUTPUT_COLUMN_OPTIONAL from pt_snap_cli.query.executor import reported_sql_limit from pt_snap_cli.query.registry import discover_categories, get_query @@ -148,6 +149,18 @@ def _import_json(result: ImportResult) -> dict[str, JsonValue]: metadata=asdict(result.metadata), focus_state=focus_state, focus_source=result.focus_state.source if result.focus_state is not None else None, + **( + { + "dataset_path": str(result.dataset_path), + "devices": list(result.devices), + "slice_count": result.slice_count, + "format": result.format, + "omitted_devices": list(result.omitted_devices), + "rebuilt": not result.reused, + } + if result.dataset_path is not None + else {} + ), ) @@ -236,7 +249,12 @@ def main( @app.command("focus") def focus_database( - db_path: Annotated[Path | None, typer.Argument(help="Path to SQLite database file")] = None, + db_path: Annotated[ + Path | None, + typer.Argument( + help="Path to SQLite database, complete manifest.json, or dataset directory" + ), + ] = None, device: Annotated[ int | None, typer.Option( @@ -257,7 +275,7 @@ def focus_database( if db_path is None and device is None: try: state = focus_service.get_focus() - except FocusFileInvalidError as e: + except (FocusFileInvalidError, DatabaseSchemaError) as e: _error_from_exc(e) if json_output: @@ -297,6 +315,7 @@ def focus_database( FocusFileInvalidError, FocusNotConfiguredError, DatabaseMissingError, + DatabaseSchemaError, InvalidDeviceError, ) as e: _error_from_exc(e) @@ -370,7 +389,23 @@ def import_snapshot( output_dir: Annotated[Path | None, typer.Option("--output-dir", "-o")] = None, device: Annotated[int | None, typer.Option("--device", "-d")] = None, no_focus: Annotated[bool, typer.Option("--no-focus", help="Skip focus update")] = False, - force: Annotated[bool, typer.Option("--force", help="Rebuild even when cache matches")] = False, + force: Annotated[ + bool, + typer.Option("--force", help="Rebuild native/single DB; never replace compatible output"), + ] = False, + events_per_slice: Annotated[ + int | None, + typer.Option( + "--events-per-slice", + help="Real events per device shard; native unless --format msinsight", + ), + ] = None, + output_format: Annotated[ + str | None, + typer.Option( + "--format", help="single-db, pt-snap-native-v2, or msinsight (compatibility-v1)" + ), + ] = None, json_output: Annotated[bool, _json_flag()] = False, ) -> None: """Import a PyTorch memory snapshot into a SQLite database.""" @@ -382,6 +417,8 @@ def import_snapshot( device=device, set_focus=not no_focus, force=force, + events_per_slice=events_per_slice, + format=output_format, ), json_output=json_output, ) @@ -389,6 +426,7 @@ def import_snapshot( ImportToolMissingError, ImportExecutionError, SnapshotFileInvalidError, + InvalidParameterError, ) as e: _error_from_exc(e) @@ -398,6 +436,10 @@ def import_snapshot( action = "Reused" if result.reused else "Imported" typer.echo(f"{action}: {result.db_path}") + if result.dataset_path is not None: + typer.echo( + f"Format: {result.format}; devices: {list(result.devices)}; slices: {result.slice_count}" + ) if result.cache_miss_reason is not None: typer.echo(f"Cache miss: {result.cache_miss_reason}") if result.focus_state is not None and result.focus_state.focus_file is not None: @@ -454,7 +496,10 @@ def split_snapshot( @app.command("metadata") def show_database_metadata( db_path: Annotated[ - Path | None, typer.Argument(help="Path to database file (optional if configured)") + Path | None, + typer.Argument( + help="Path to database, manifest.json, or dataset directory (optional if configured)" + ), ] = None, json_output: Annotated[bool, _json_flag()] = False, ) -> None: @@ -491,7 +536,10 @@ def show_database_metadata( @app.command("query") def query_database( db_path: Annotated[ - Path | None, typer.Argument(help="Path to database file (optional if configured)") + Path | None, + typer.Argument( + help="Path to database, manifest.json, or dataset directory (optional if configured)" + ), ] = None, template_use: Annotated[ str | None, @@ -509,6 +557,10 @@ def query_database( autocompletion=complete_device_ids, ), ] = None, + slice_index: Annotated[ + int | None, + typer.Option("--slice", help="Explicit dataset slice index for event inspection"), + ] = None, list_templates: Annotated[ bool, typer.Option("--list", help="List all available query templates") ] = False, @@ -613,6 +665,7 @@ def query_database( typer.echo(f"Description: {info.description}") typer.echo(f"Category: {info.category}") typer.echo(f"Devices: {info.devices}") + typer.echo(f"Dataset Support: {json.dumps(info.dataset_support, sort_keys=True)}") typer.echo( "Semantics Version: " f"{info.semantics_version if info.semantics_version is not None else 'none'}" @@ -702,6 +755,7 @@ def query_database( max_rows=max_rows, exact_total=exact_total, timeout_s=timeout, + slice_index=slice_index, ) if json_output: resolved = focus_service.resolve_focus( @@ -725,6 +779,7 @@ def query_database( total_is_exact=result.total_is_exact, timeout_s=result.timeout_s, rows=result.rows, + **({"scope": result.scope} if result.scope is not None else {}), ) ) return @@ -779,6 +834,7 @@ def query_database( except ( TemplateRenderError, QueryExecutionError, + InvalidParameterError, DatabaseSchemaError, json.JSONDecodeError, ) as e: @@ -860,7 +916,10 @@ def show_capabilities( @app.command("overview") def show_database_overview( db_path: Annotated[ - Path | None, typer.Argument(help="Path to database file (optional if configured)") + Path | None, + typer.Argument( + help="Path to database, manifest.json, or dataset directory (optional if configured)" + ), ] = None, json_output: Annotated[bool, _json_flag()] = False, ) -> None: @@ -882,6 +941,12 @@ def show_database_overview( _emit_json(json_success(**service.overview_to_dict(overview))) return _print_database_overview(overview) + if overview.dataset is not None: + typer.echo(f"Dataset: {overview.dataset['format']} ({overview.dataset['status']})") + typer.echo(f"Slices/devices: {overview.dataset['devices']}") + typer.echo( + "Text callstacks available; structured frames and cross-slice queries unavailable." + ) # Flag-only options used when a parse failure happens before the --json callback. diff --git a/src/pt_snap_cli/cli_reports.py b/src/pt_snap_cli/cli_reports.py index b9abd25..f20831f 100644 --- a/src/pt_snap_cli/cli_reports.py +++ b/src/pt_snap_cli/cli_reports.py @@ -29,6 +29,7 @@ FocusNotConfiguredError, FocusService, InvalidDeviceError, + InvalidParameterError, QueryExecutionError, ReportService, TemplateNotFoundError, @@ -71,6 +72,16 @@ def report_peak_memory( help="Opt-in UTF-8 byte budget per callstack (0: identity only; negative: full text). Not a response byte cap or token budget.", ), ] = -1, + start_id: Annotated[ + int | None, typer.Option("--start-id", help="First real event ID in peak range") + ] = None, + end_id: Annotated[ + int | None, typer.Option("--end-id", help="Last real event ID in peak range") + ] = None, + timeout: Annotated[ + float | None, + typer.Option("--timeout", help="One report-wide deadline in seconds; <=0 disables"), + ] = None, json_output: Annotated[bool, _json_flag()] = False, ) -> None: """Generate a peak memory attribution report.""" @@ -84,6 +95,9 @@ def report_peak_memory( include_static=include_static, limit=limit, stack_bytes=stack_bytes, + start_id=start_id, + end_id=end_id, + timeout_s=timeout, ) except ValueError as e: _error(str(e), code=INVALID_PARAMETER, hint="Use --metric active, allocated, or reserved.") @@ -120,6 +134,7 @@ def report_peak_memory( except ( TemplateNotFoundError, TemplateRenderError, + InvalidParameterError, QueryExecutionError, DatabaseSchemaError, ) as e: diff --git a/src/pt_snap_cli/completion.py b/src/pt_snap_cli/completion.py index abbd9cc..797dfff 100644 --- a/src/pt_snap_cli/completion.py +++ b/src/pt_snap_cli/completion.py @@ -75,6 +75,11 @@ def complete_device_ids() -> list[str]: return [] try: + from pt_snap_cli.core.dataset_resolver import DatasetResolver + + dataset = DatasetResolver().inspect(db_path) + if dataset is not None: + return [str(device) for device in dataset.device_ids] uri = f"file:{db_path}?mode=ro" with closing(sqlite3.connect(uri, uri=True)) as conn: cursor = conn.cursor() diff --git a/src/pt_snap_cli/context.py b/src/pt_snap_cli/context.py index c64b772..140eca4 100644 --- a/src/pt_snap_cli/context.py +++ b/src/pt_snap_cli/context.py @@ -44,12 +44,17 @@ def __init__( devices: list[int] | None = None, *, persistent: bool = False, + immutable: bool = False, ): """Initialize context with database path. Args: db_path: Path to the SQLite database file. devices: Optional list of device IDs to filter. + immutable: Internal opt-in for a closed, finalized compatibility + dataset after ALL member/parent/sidecar checks. Never use this + to read a live WAL or as a concurrency/sandbox guarantee. The + standalone default remains mode=ro without immutable. persistent: When True, the connection stays open after each ``connect()`` call so it can be reused. The caller is responsible for invoking ``close()`` to release it. @@ -61,6 +66,9 @@ def __init__( self.db_path = Path(db_path) self._devices = devices self._persistent = persistent + # Opt-in only after dataset-wide finalized-member checks. Standalone + # analysis keeps mode=ro without immutable and its existing WAL behavior. + self._immutable = immutable self._conn: sqlite3.Connection | None = None self._connect_depth = 0 self._device_ids: list[int] | None = None @@ -204,7 +212,8 @@ def discover_devices(self) -> list[int]: def device_trace_bounds(self) -> list[tuple[int, int | None, int | None]]: """Return ``(device_id, first_event_id, last_event_id)`` for each device. - Bounds come from ``MIN(id)`` / ``MAX(id)`` on ``trace_entry_``. + Bounds come from nonnegative real IDs on ``trace_entry_``. + Negative synthetic boundary rows do not extend the real trace. Empty tables report ``None`` for both ends. The connection stays read-only (``mode=ro``). """ @@ -213,7 +222,7 @@ def device_trace_bounds(self) -> list[tuple[int, int | None, int | None]]: cursor = conn.cursor() for device_id in self.device_ids: table = f"trace_entry_{device_id}" - cursor.execute(f"SELECT MIN(id), MAX(id) FROM {_quote_ident(table)}") + cursor.execute(f"SELECT MIN(id), MAX(id) FROM {_quote_ident(table)} WHERE id >= 0") row = cursor.fetchone() first = int(row[0]) if row is not None and row[0] is not None else None last = int(row[1]) if row is not None and row[1] is not None else None @@ -259,6 +268,8 @@ def connect(self) -> Generator[sqlite3.Connection, None, None]: # Keep the colon in Windows drive prefixes while encoding SQLite URI metacharacters. encoded_path = quote(self.db_path.as_posix(), safe="/:") uri = f"file:{encoded_path}?mode=ro" + if self._immutable: + uri += "&immutable=1" self._conn = sqlite3.connect(uri, uri=True) self._conn.row_factory = sqlite3.Row diff --git a/src/pt_snap_cli/core/context_cache.py b/src/pt_snap_cli/core/context_cache.py index 296c6e0..c345707 100644 --- a/src/pt_snap_cli/core/context_cache.py +++ b/src/pt_snap_cli/core/context_cache.py @@ -39,7 +39,9 @@ def __init__(self, maxsize: int = 4) -> None: raise ValueError("maxsize must be positive") self._maxsize = maxsize # Insertion order doubles as LRU recency (oldest first). - self._entries: OrderedDict[Path, tuple[Context, FileSignature]] = OrderedDict() + self._entries: OrderedDict[Path, tuple[Context, tuple[FileSignature, str | None, bool]]] = ( + OrderedDict() + ) @property def maxsize(self) -> int: @@ -48,7 +50,9 @@ def maxsize(self) -> int: def __len__(self) -> int: return len(self._entries) - def get(self, db_path: Path | str) -> Context: + def get( + self, db_path: Path | str, *, generation: str | None = None, immutable: bool = False + ) -> Context: """Return a cached or freshly opened :class:`Context` for ``db_path``. On a hit the cached entry is promoted to most-recently-used. On a @@ -65,7 +69,9 @@ def get(self, db_path: Path | str) -> Context: # Probe metadata first so a missing/inaccessible file raises # consistently whether the cache is hit or missed. - current_signature = self._file_signature(key) + # A dataset reader can supply its validated manifest/member content identity. + # Single-DB callers retain the original filesystem-signature behavior. + current_signature = (self._file_signature(key), generation, immutable) cached = self._entries.get(key) if cached is not None: @@ -76,7 +82,11 @@ def get(self, db_path: Path | str) -> Context: self._safe_close(ctx) del self._entries[key] - ctx = Context(key, persistent=True) + ctx = ( + Context(key, persistent=True, immutable=True) + if immutable + else Context(key, persistent=True) + ) self._entries[key] = (ctx, current_signature) self._evict_if_needed() return ctx diff --git a/src/pt_snap_cli/core/dataset_attribution.py b/src/pt_snap_cli/core/dataset_attribution.py new file mode 100644 index 0000000..0fcc5a0 --- /dev/null +++ b/src/pt_snap_cli/core/dataset_attribution.py @@ -0,0 +1,263 @@ +"""Dataset point-event lifecycle queries using the shared batched source layer.""" + +from __future__ import annotations + +from typing import cast + +from pt_snap_cli.core.dataset_sources import DatasetSourceResolver, EventSource +from pt_snap_cli.core.models import QueryResult +from pt_snap_cli.core.stack_summary import summarize_stacks + +STATIC_LABEL = "[static] allocEventId=-1, freeEventId=-1" +PREEXISTING_LABEL = "[preexisting live] allocEventId=-1" +MISSING_LABEL = "[missing callstack]" + + +def event_attribution( + sources: DatasetSourceResolver, + template: str, + params: dict[str, object], + max_rows: int | None, + exact_total: bool, + slice_index: int | None, + semantics_version: int | None, +) -> QueryResult: + event_id = cast(int, params["event_id"]) + item = sources.target(event_id, slice_index) + path = sources.dataset.root / item.file + blocks = sources.read( + path, + f'SELECT * FROM "block_{sources.device.device_id}" WHERE size>=? ' + "AND (allocEventId=-1 OR allocEventId<=?) AND (freeEventId=-1 OR freeEventId>?) " + + ("AND allocEventId!=-1 " if not params["include_static"] else "") + + "ORDER BY id", + [params["min_size"], event_id, event_id], + ) + references = sources.events( + cast(int, block[field]) + for block in blocks + for field in ("allocEventId", "freeEventId") + if cast(int, block[field]) >= 0 + ) + allocation_events = allocation_stacks = free_events = ordered_blocks = dynamic_bytes = ( + captured_bytes + ) = 0 + dynamic_blocks = preexisting_blocks = 0 + for block in blocks: + sources.budget.remaining() + alloc, free = cast(int, block["allocEventId"]), cast(int, block["freeEventId"]) + category = ( + "dynamic_live_at_event" + if alloc >= 0 + else "static" if free == -1 else "preexisting_live_at_event" + ) + block["category"] = category + block["lifecycle_id"] = f"{sources.namespace}:" + ( + f"alloc:{alloc}" if alloc >= 0 else f"preexisting:{block['id']}" + ) + # The stored state is a replay/slice observation, not necessarily state at E. + block["state_scope"] = "slice_observation" + allocation, completion = references.get(alloc), references.get(free) + # Base schemas can be structurally valid yet have an invalid source action. + # Preserve bytes/category and report the missing proof instead of guessing. + allocation = ( + allocation if allocation is not None and allocation.event["action"] == 4 else None + ) + completion = ( + completion if completion is not None and completion.event["action"] == 6 else None + ) + block["allocation_source"] = allocation.to_dict() if allocation else None + block["free_source"] = completion.to_dict() if completion else None + block["allocation_source_status"] = ( + "unknown_preexisting" if alloc == -1 else "resolved" if allocation else "missing" + ) + block["free_source_status"] = ( + "live_or_unknown" if free == -1 else "resolved" if completion else "missing" + ) + if alloc >= 0: + dynamic_blocks += 1 + dynamic_bytes += cast(int, block["size"]) + allocation_events += allocation is not None + allocation_stacks += allocation is not None and allocation.stack_kind == "captured" + ordered_blocks += allocation is not None and allocation.frames is not None + if allocation is not None and allocation.stack_kind == "captured": + captured_bytes += cast(int, block["size"]) + else: + preexisting_blocks += 1 + free_events += completion is not None + coverage: dict[str, object] = { + "range_complete": True, + "event_position": "after", + "active_blocks": len(blocks), + "active_bytes": sum(cast(int, b["size"]) for b in blocks), + "dynamic_blocks": dynamic_blocks, + "dynamic_bytes": dynamic_bytes, + "allocation_events_resolved": allocation_events, + "allocation_stacks_captured": allocation_stacks, + "allocation_source_complete": allocation_events == dynamic_blocks, + "captured_dynamic_bytes": captured_bytes, + "unknown_preexisting_blocks": preexisting_blocks, + "free_references": sum(cast(int, b["freeEventId"]) >= 0 for b in blocks), + "free_events_resolved": free_events, + "ordered_frame_blocks": ordered_blocks, + "ordered_frames_complete": dynamic_blocks > 0 and ordered_blocks == dynamic_blocks, + "frame_degradation": ( + None + if dynamic_blocks > 0 and ordered_blocks == dynamic_blocks + else "text_only_or_uncovered" + ), + "source_queries": sources.query_count, + } + scope: dict[str, object] = { + "kind": "event", + "device_id": sources.device.device_id, + "slice_index": item.index, + "first_event_id": event_id, + "last_event_id": event_id, + "db_path": str(path), + "dataset_path": str(sources.dataset.root), + "fingerprint": sources.dataset.fingerprint, + "boundary_events_included": False, + "source_coverage": coverage, + } + if template == "active_blocks_at_event": + key = cast(str, params["order_by"]) + blocks.sort(key=lambda b: cast(int, b["id"])) + blocks.sort(key=lambda b: cast(int, b[key]), reverse=params["order_dir"] == "DESC") + all_rows = blocks + rank_full = False + limit = cast(int, params["limit"]) + offset = max(0, cast(int, params["offset"])) + selected = all_rows[offset : offset + limit] if limit >= 0 else all_rows[offset:] + else: + all_rows = _groups(sources, blocks, references) + top_n = cast(int, params["top_n"]) + dynamic = [row for row in all_rows if row["category"] == "dynamic_live_at_event"] + rank_full = top_n >= 0 and bool(dynamic) and len(dynamic) >= top_n + selected = [row for row in all_rows if row["category"] != "dynamic_live_at_event"] + ( + dynamic[:top_n] if top_n >= 0 else dynamic + ) + selected.sort(key=_rank) + denominator = sum(cast(int, row["size_bytes"]) for row in selected) + for row in selected: + row["percent_of_active_blocks"] = ( + round(cast(int, row["size_bytes"]) * 100.0 / denominator, 4) + if denominator + else None + ) + offset = 0 + has_more = len(selected) < len(all_rows) - offset or rank_full + if max_rows is not None and max_rows > 0: + has_more = has_more or len(selected) > max_rows + selected = selected[:max_rows] + if exact_total: + has_more = len(all_rows) > len(selected) + offset + sources.budget.consume(selected) + return QueryResult( + total=len(all_rows) if exact_total else len(selected), + returned=len(selected), + device_id=sources.device.device_id, + rows=selected, + template=template, + semantics_version=semantics_version, + has_more=has_more, + truncated=has_more or offset > 0, + total_is_exact=exact_total or not has_more and offset == 0, + timeout_s=sources.budget.timeout_s, + scope=scope, + ) + + +def _rank(row: dict[str, object]) -> tuple[int, int, str, str]: + return ( + -cast(int, row["size_bytes"]), + -cast(int, row["block_count"]), + cast(str, row["callstack"]), + cast(str, row["source_stack_id"]), + ) + + +def _groups( + sources: DatasetSourceResolver, + blocks: list[dict[str, object]], + references: dict[int, EventSource], +) -> list[dict[str, object]]: + grouped: dict[str, dict[str, object]] = {} + for block in blocks: + sources.budget.remaining() + category = cast(str, block["category"]) + source = references.get(cast(int, block["allocEventId"])) + if block["allocation_source"] is None: + source = None + kind = source.stack_kind if source is not None else "missing" + if category != "dynamic_live_at_event": + kind = category + sourced = ( + source is not None + and category == "dynamic_live_at_event" + and (kind == "captured" or source.frames is not None) + ) + identity = ( + source.stack_id + if sourced and source is not None + else f"{sources.namespace}:category:{kind}" + ) + text = ( + source.event["callstack"] + if kind == "captured" and source is not None and source.text_kind == "captured" + else ( + STATIC_LABEL + if kind == "static" + else PREEXISTING_LABEL if kind == "preexisting_live_at_event" else MISSING_LABEL + ) + ) + group = grouped.setdefault( + identity, + { + "callstack": text, + "category": category, + "block_count": 0, + "size_bytes": 0, + "requested_bytes": 0, + "source_stack_id": identity, + "stack_kind": kind, + "text_kind": source.text_kind if sourced and source else kind, + "stack_id": identity, + "stack_event_id": source.event["id"] if sourced and source else None, + "frames": source.frames if sourced and source else None, + "frames_status": (source.frames_status if sourced and source else "text_only"), + }, + ) + group["block_count"] = cast(int, group["block_count"]) + 1 + group["size_bytes"] = cast(int, group["size_bytes"]) + cast(int, block["size"]) + group["requested_bytes"] = cast(int, group["requested_bytes"]) + cast( + int, block["requestedSize"] + ) + if source is not None and sourced: + if group["text_kind"] != "captured" and source.text_kind == "captured": + group["callstack"] = source.event["callstack"] + group["text_kind"] = "captured" + group["stack_event_id"] = source.event["id"] + elif group["text_kind"] != "captured" or source.text_kind == "captured": + group["stack_event_id"] = min( + cast(int, group["stack_event_id"]), cast(int, source.event["id"]) + ) + rows = list(grouped.values()) + for row in rows: + row["size_gib"] = round(cast(int, row["size_bytes"]) / 1073741824.0, 6) + row["requested_gib"] = round(cast(int, row["requested_bytes"]) / 1073741824.0, 6) + rows.sort(key=_rank) + return rows + + +def summarize_dataset_stacks(rows: list[dict[str, object]], budget: int) -> None: + """Shorten display only; retain the already proved cross-shard identity.""" + identities = [(row["stack_id"], row["stack_kind"]) for row in rows] + for row in rows: + # Raw frames may be captured even when the formatted TEXT is absent. + if not isinstance(row.get("callstack"), str): + row["stack_kind"] = "missing" + summarize_stacks(rows, budget, "v1") + for row, (identity, kind) in zip(rows, identities, strict=True): + row["stack_id"] = identity + row["stack_kind"] = kind diff --git a/src/pt_snap_cli/core/dataset_contract.py b/src/pt_snap_cli/core/dataset_contract.py new file mode 100644 index 0000000..9d8be4c --- /dev/null +++ b/src/pt_snap_cli/core/dataset_contract.py @@ -0,0 +1,502 @@ +"""P0 sharded SnapshotDB contract, independent of import/query execution. + +Original pt-snap implementation of the documented interoperability facts at +Ascend/msinsight revision 101f65b877a267ffd5f66ea3834706057ba243e5. +No upstream implementation is copied. See docs/en/sharded-snapshotdb.md. +""" + +from __future__ import annotations + +import json +import re +import sqlite3 +from collections.abc import Iterable +from contextlib import closing +from dataclasses import dataclass +from pathlib import Path +from typing import Literal, cast + +from .dataset_files import require_readonly_member +from .errors import DatabaseSchemaError + +MANIFEST_SCHEMA_VERSION = 1 +MSINSIGHT_REVISION = "101f65b877a267ffd5f66ea3834706057ba243e5" +ACTION_NAMES = ( + "segment_map", + "segment_unmap", + "segment_alloc", + "segment_free", + "alloc", + "free_requested", + "free_completed", + "workspace_snapshot", +) +BLOCK_STATES = ((-1, "inactive"), (0, "active_pending_free"), (1, "active_allocated")) +TRACE_COLUMNS = tuple( + (name, "INTEGER") + for name in ("id", "action", "address", "size", "stream", "allocated", "active", "reserved") +) + (("callstack", "TEXT"),) +BLOCK_COLUMNS = tuple( + (name, "INTEGER") + for name in ("id", "address", "size", "requestedSize", "state", "allocEventId", "freeEventId") +) +DICTIONARY_COLUMNS = tuple((name, "TEXT") for name in ("table", "column", "key", "value")) + + +class DatasetContractError(ValueError): + """A validation failure with a stable code and manifest/file location.""" + + def __init__(self, location: str, code: str, detail: str): + self.location = location + self.code = code + super().__init__(f"{location}: {code}: {detail}") + + +def _need(condition: bool, location: str, code: str, detail: str) -> None: + if not condition: + raise DatasetContractError(location, code, detail) + + +def _object(value: object, location: str) -> dict[str, object]: + _need(isinstance(value, dict), location, "type", "expected object") + return cast(dict[str, object], value) + + +def _array(value: object, location: str) -> list[object]: + _need(isinstance(value, list), location, "type", "expected array") + return cast(list[object], value) + + +def _integer(value: object, location: str, minimum: int = 0, maximum: int = 2**63 - 1) -> int: + _need(type(value) is int, location, "type", "expected integer (not boolean)") + number = cast(int, value) + _need(minimum <= number <= maximum, location, "range", "integer out of range") + return number + + +def _text(value: object, location: str, *, empty: bool = False) -> str: + _need(isinstance(value, str), location, "type", "expected string") + text = cast(str, value) + _need(empty or bool(text), location, "range", "string must not be empty") + return text + + +@dataclass(frozen=True) +class SliceRecord: + index: int + start_event_id: int + end_event_id: int + file: str + ready: bool + + +@dataclass(frozen=True) +class DeviceRecord: + device_id: int + event_count: int + ready_slices: tuple[int, ...] + slices: tuple[SliceRecord, ...] + + +@dataclass(frozen=True) +class DatasetManifest: + schema_version: int + status: str + source_file: str + cache_hash: str + events_per_slice: int + devices: tuple[DeviceRecord, ...] + + def require_complete(self) -> None: + _need(self.status == "complete", "status", "building", "complete dataset required") + for device in self.devices: + _need( + all(item.ready for item in device.slices), + f"devices.{device.device_id}.readySlices", + "not_ready", + "complete requires every slice ready after finalization", + ) + + +def parse_manifest(value: object, *, require_complete: bool = True) -> DatasetManifest: + """Validate JSON-shaped data; inspection may explicitly accept building. + + Unknown extension declarations are ignored, never promoted to capabilities. + This does not check files; use validate_dataset for finalized artifacts. + """ + data = _object(value, "manifest") + version = _integer(data.get("schemaVersion"), "schemaVersion", maximum=2**31 - 1) + _need( + version == MANIFEST_SCHEMA_VERSION, "schemaVersion", "version", "unsupported base version" + ) + status = _text(data.get("status"), "status") + _need(status in ("building", "complete"), "status", "status", "unknown publication state") + source = _text(data.get("sourceFile"), "sourceFile") + cache_hash = _text(data.get("cacheHash"), "cacheHash", empty=True) + capacity = _integer(data.get("eventsPerSlice"), "eventsPerSlice", minimum=1) + devices_data = _object(data.get("devices"), "devices") + _need(bool(devices_data), "devices", "empty", "at least one event-bearing device required") + devices: list[DeviceRecord] = [] + for key, value in devices_data.items(): + location = f"devices.{key}" + _need( + isinstance(key, str) and re.fullmatch(r"0|[1-9][0-9]*", key) is not None, + location, + "device", + "expected canonical nonnegative device ID", + ) + device_id = _integer(int(key), location, maximum=2**31 - 1) + device = _object(value, location) + count = _integer(device.get("eventCount"), location + ".eventCount", minimum=1) + size = _integer( + device.get("sliceCount"), location + ".sliceCount", minimum=1, maximum=2**31 - 1 + ) + ready = tuple( + _integer(item, location + ".readySlices", maximum=size - 1) + for item in _array(device.get("readySlices"), location + ".readySlices") + ) + _need( + len(set(ready)) == len(ready), + location + ".readySlices", + "duplicate", + "duplicate ready index", + ) + records = _array(device.get("slices"), location + ".slices") + _need( + len(records) == size, + location + ".sliceCount", + "count", + "sliceCount differs from slices", + ) + slices: list[SliceRecord] = [] + expected_start = 0 + for index, raw in enumerate(records): + loc = f"{location}.slices[{index}]" + item = _object(raw, loc) + number = _integer(item.get("index"), loc + ".index", maximum=2**31 - 1) + _need(number == index, loc + ".index", "index", "indices must match array order") + start = _integer(item.get("startEventId"), loc + ".startEventId") + end = _integer(item.get("endEventId"), loc + ".endEventId", minimum=start) + _need( + start == expected_start, loc, "coverage", "ranges must cover continuously from zero" + ) + _need(end - start + 1 <= capacity, loc, "capacity", "slice exceeds eventsPerSlice") + file = _text(item.get("file"), loc + ".file") + _need( + not any(part in ("", ".", "..") for part in file.split("/")) + and not any(char in file for char in ("\\", ":", "\0")), + loc + ".file", + "path", + "unsafe relative path", + ) + _need( + file == f"device_{device_id}/slice_{index:05d}.db", + loc + ".file", + "path", + "file must match device and slice index", + ) + is_ready = item.get("ready") + _need(type(is_ready) is bool, loc + ".ready", "type", "expected boolean") + _need( + is_ready == (index in ready), + loc + ".ready", + "readiness", + "ready disagrees with readySlices", + ) + slices.append(SliceRecord(index, start, end, file, cast(bool, is_ready))) + expected_start = end + 1 + _need( + expected_start == count, + location + ".eventCount", + "coverage", + "ranges do not cover eventCount", + ) + devices.append(DeviceRecord(device_id, count, ready, tuple(slices))) + manifest = DatasetManifest(version, status, source, cache_hash, capacity, tuple(devices)) + # Even inspection must reject a contradictory complete publication. + if require_complete or status == "complete": + manifest.require_complete() + return manifest + + +def validate_event_ids(event_ids: Iterable[int]) -> int: + """Producer precondition: actual real-event positions, not max(id)+1. + + Boundary rows are a separate input category. No renumbering is performed. + """ + count = 0 + for position, value in enumerate(event_ids): + number = _integer(value, f"events[{position}]") + _need( + number == position, + f"events[{position}]", + "event_identity", + "compatibility requires original real IDs 0..N-1; do not silently renumber", + ) + count += 1 + _need(count > 0, "events", "empty", "empty/static-only devices cannot use compatibility v1") + return count + + +@dataclass(frozen=True) +class QueryScope: + """Selector contract only; not a cross-slice query executor.""" + + kind: Literal["dataset", "device", "slice", "event_range"] + device_id: int | None = None + slice_index: int | None = None + start_event_id: int | None = None + end_event_id: int | None = None + + def validate(self, manifest: DatasetManifest) -> None: + manifest.require_complete() + loc = "scope" + _need( + self.kind in ("dataset", "device", "slice", "event_range"), loc, "kind", "unknown scope" + ) + if self.kind == "dataset": + _need( + all( + value is None + for value in ( + self.device_id, + self.slice_index, + self.start_event_id, + self.end_event_id, + ) + ), + loc, + "selector", + "dataset has no selectors", + ) + return + device_id = _integer(self.device_id, loc + ".device_id", maximum=2**31 - 1) + device = next((item for item in manifest.devices if item.device_id == device_id), None) + _need(device is not None, loc, "device", "unknown device") + assert device is not None + if self.kind == "slice": + _integer(self.slice_index, loc + ".slice_index", maximum=len(device.slices) - 1) + else: + _need( + self.slice_index is None, loc, "selector", "slice index only valid for slice scope" + ) + if self.kind == "event_range": + start = _integer( + self.start_event_id, loc + ".start_event_id", maximum=device.event_count - 1 + ) + _integer( + self.end_event_id, + loc + ".end_event_id", + minimum=start, + maximum=device.event_count - 1, + ) + else: + _need( + self.start_event_id is None and self.end_event_id is None, + loc, + "selector", + "unexpected range", + ) + + +@dataclass(frozen=True) +class DatasetValidation: + manifest: DatasetManifest + real_event_count: int + boundary_event_count: int + text_callstacks: bool = True + structured_frames: bool = False + query_execution: bool = False + + +def _unique_object(pairs: list[tuple[str, object]]) -> dict[str, object]: + result: dict[str, object] = {} + for key, value in pairs: + _need(key not in result, "manifest." + key, "duplicate", "duplicate JSON member") + result[key] = value + return result + + +def _safe_file(root: Path, relative: str) -> Path: + path = root / relative + for part in ( + root, + *[ + root.joinpath(*Path(relative).parts[:i]) + for i in range(1, len(Path(relative).parts) + 1) + ], + ): + _need(not part.is_symlink(), relative, "path", "symlink is not an artifact member") + _need(path.is_file(), relative, "missing", "artifact file missing") + return path + + +def _schema( + conn: sqlite3.Connection, table: str, columns: tuple[tuple[str, str], ...], loc: str +) -> None: + kind = conn.execute("SELECT type FROM sqlite_master WHERE name=?", (table,)).fetchone() + _need(kind == ("table",), loc, "table", f"{table} must be an actual table, not a view") + rows = conn.execute(f'PRAGMA table_info("{table}")').fetchall() + _need( + tuple((row[1], row[2].upper()) for row in rows) == columns, + loc, + "columns", + f"{table} column order/types differ from base contract", + ) + if table != "dictionary": + _need( + tuple(row[5] for row in rows) == (1,) + (0,) * (len(rows) - 1), + loc, + "primary_key", + f"{table}.id must be the sole primary key", + ) + + +def _validate_slice( + conn: sqlite3.Connection, + device: DeviceRecord, + item: SliceRecord, + identities: dict[tuple[int, int], tuple[object, ...]], +) -> int: + loc = item.file + trace, block = f"trace_entry_{device.device_id}", f"block_{device.device_id}" + tables = { + row[0] + for row in conn.execute("SELECT name FROM sqlite_master WHERE type IN ('table','view')") + } + relevant = {name for name in tables if name.lower().startswith(("trace_entry_", "block_"))} + _need(relevant == {trace, block}, loc, "device_tables", "device/table set mismatch") + _need( + "callstack" not in {name.lower() for name in tables}, + loc, + "layout", + "v1 must not contain native v2 callstack table", + ) + for table, columns in ( + (trace, TRACE_COLUMNS), + (block, BLOCK_COLUMNS), + ("dictionary", DICTIONARY_COLUMNS), + ): + _schema(conn, table, columns, loc) + for table, column, values in ( + (trace, "action", tuple(enumerate(ACTION_NAMES))), + (block, "state", BLOCK_STATES), + ): + rows = conn.execute( + 'SELECT "key", "value" FROM dictionary WHERE "table"=? AND "column"=?', (table, column) + ).fetchall() + expected = {(str(key), name) for key, name in values} + _need( + len(rows) == len(expected) and set(rows) == expected, + loc, + "dictionary", + f"invalid {table}.{column} mapping", + ) + count, low, high = conn.execute( + f'SELECT COUNT(*), MIN(id), MAX(id) FROM "{trace}" WHERE id>=0' + ).fetchone() + _need( + (count, low, high) + == (item.end_event_id - item.start_event_id + 1, item.start_event_id, item.end_event_id), + loc, + "event_identity", + "real rows differ from declared contiguous interval", + ) + bad = conn.execute( + f'SELECT id FROM "{trace}" WHERE action NOT BETWEEN 0 AND 7 OR action IS NULL ' + "OR typeof(action) != 'integer' OR (id<0 AND action NOT IN (0,2)) LIMIT 1" + ).fetchone() + _need(bad is None, loc, "event_action", "unknown action or invalid boundary event") + invalid_types = " OR ".join(f"typeof({name}) != 'integer'" for name, _ in TRACE_COLUMNS[:-1]) + bad = conn.execute( + f'SELECT id FROM "{trace}" WHERE {invalid_types} ' + "OR size<0 OR allocated<0 OR active<0 OR reserved<0 " + "OR typeof(callstack) NOT IN ('text','null') LIMIT 1" + ).fetchone() + _need(bad is None, loc, "event_values", "invalid scalar type or negative byte metric") + boundary_count = conn.execute(f'SELECT COUNT(*) FROM "{trace}" WHERE id<0').fetchone()[0] + for row in conn.execute(f'SELECT * FROM "{block}"'): + block_id, address, size, requested, state, alloc, free = row + for name, value in zip( + ("id", "address", "size", "requestedSize", "state", "allocEventId", "freeEventId"), + row, + strict=True, + ): + _integer(value, f"{loc}.{block}.{block_id}.{name}", minimum=-(2**63)) + _need( + state in (-1, 0, 1) and size >= 0 and 0 <= requested <= size, + loc, + "block", + "invalid state/size", + ) + _need( + (alloc == -1 and block_id < 0) + or (0 <= alloc < device.event_count and block_id == alloc), + loc, + "block_identity", + "resolved ID must equal allocation; unresolved ID must be negative", + ) + _need( + free == -1 or (0 <= free < device.event_count and (alloc == -1 or free >= alloc)), + loc, + "lifecycle", + "invalid free sentinel or lifecycle order", + ) + identity = (device.device_id, block_id) + # State describes the slice observation and may differ; identity/lifetime do not. + signature = (address, size, requested, alloc, free) + _need( + identity not in identities or identities[identity] == signature, + loc, + "block_identity", + "same cross-slice ID has conflicting identity/lifetime", + ) + identities[identity] = signature + return cast(int, boundary_count) + + +def validate_dataset(directory: Path | str) -> DatasetValidation: + """Read a complete compatibility-v1 dataset without migration or pickle. + + Checks are validation, not a filesystem sandbox or concurrent publication lock. + Missing pt_snap_metadata/extension tables are valid external base artifacts. + """ + root = Path(directory).absolute() + _need( + root.resolve() == root and not root.is_symlink(), + "dataset", + "path", + "canonical non-symlink directory required", + ) + try: + path = _safe_file(root, "manifest.json") + with path.open(encoding="utf-8") as source: + data: object = json.load(source, object_pairs_hook=_unique_object) + manifest = parse_manifest(data) + # Preflight ALL members before the first SQLite open, also for callers + # using this bare P0 validator rather than DatasetResolver. + for device in manifest.devices: + for item in device.slices: + member = _safe_file(root, item.file) + try: + require_readonly_member(root, member, immutable=True) + except DatabaseSchemaError as exc: + raise DatasetContractError(item.file, "transport", str(exc)) from exc + identities: dict[tuple[int, int], tuple[object, ...]] = {} + boundaries = 0 + for device in manifest.devices: + for item in device.slices: + db_path = _safe_file(root, item.file) + try: + with closing( + sqlite3.connect(db_path.as_uri() + "?mode=ro&immutable=1", uri=True) + ) as conn: + conn.execute("PRAGMA query_only=ON") + boundaries += _validate_slice(conn, device, item, identities) + except sqlite3.DatabaseError as exc: + raise DatasetContractError(item.file, "sqlite", str(exc)) from exc + return DatasetValidation( + manifest, sum(item.event_count for item in manifest.devices), boundaries + ) + except (OSError, UnicodeError, json.JSONDecodeError) as exc: + raise DatasetContractError("manifest.json", "read", str(exc)) from exc diff --git a/src/pt_snap_cli/core/dataset_files.py b/src/pt_snap_cli/core/dataset_files.py new file mode 100644 index 0000000..e32a6d6 --- /dev/null +++ b/src/pt_snap_cli/core/dataset_files.py @@ -0,0 +1,35 @@ +"""Shared pre-open dataset checks. No repair, migration, WAL changes or SQLite opens.""" + +from __future__ import annotations + +import hashlib +from pathlib import Path + +from .errors import DatabaseSchemaError + + +def require_readonly_member(root: Path, member: Path, *, immutable: bool = False) -> None: + parts = member.relative_to(root).parts + for index in range(1, len(parts) + 1): + if root.joinpath(*parts[:index]).is_symlink(): + raise DatabaseSchemaError(f"Dataset member must not use symlink paths: {member}") + if member.resolve() != member or not member.is_file(): + raise DatabaseSchemaError(f"Dataset member is missing or not a canonical file: {member}") + for suffix in ("-wal", "-journal", "-shm"): + sidecar = Path(str(member) + suffix) + if sidecar.exists() or sidecar.is_symlink(): + raise DatabaseSchemaError(f"Dataset member has a live SQLite sidecar: {member}") + with member.open("rb") as source: + header = source.read(20) + # Only finalized compatibility datasets use immutable transport. Native + # datasets retain their non-WAL contract; no writer repair/checkpoint occurs. + if not immutable and header[:16] == b"SQLite format 3\x00" and 2 in header[18:20]: + raise DatabaseSchemaError(f"Dataset member uses persistent WAL mode: {member}") + + +def hash_file(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as source: + for chunk in iter(lambda: source.read(1024 * 1024), b""): + digest.update(chunk) + return digest.hexdigest() diff --git a/src/pt_snap_cli/core/dataset_global.py b/src/pt_snap_cli/core/dataset_global.py new file mode 100644 index 0000000..cbc73f6 --- /dev/null +++ b/src/pt_snap_cli/core/dataset_global.py @@ -0,0 +1,484 @@ +"""Explicit built-in global algorithms; no local caps or temporary merge database. + +Fetched work is cumulatively bounded by QueryBudget, including sources and final +output. Sorting/materialization is O(bounded work), NOT O(max_rows). A budget +failure rejects the operation rather than returning an alleged global answer. +""" + +from __future__ import annotations + +import hashlib +from typing import cast + +from pt_snap_cli.core.dataset_contract import QueryScope, SliceRecord +from pt_snap_cli.core.dataset_sources import DatasetSourceResolver +from pt_snap_cli.core.errors import InvalidParameterError, QueryExecutionError +from pt_snap_cli.core.models import QueryResult + +METRICS = ("allocated", "active", "reserved") + + +def _range( + sources: DatasetSourceResolver, params: dict[str, object], slice_index: int | None +) -> tuple[int, int, list[SliceRecord]]: + device = sources.device + if slice_index is not None: + sources.dataset.paths(QueryScope("slice", device.device_id, slice_index)) + members = [device.slices[slice_index]] + else: + members = list(device.slices) + low, high = members[0].start_event_id, members[-1].end_event_id + start = params.get("start_id", params.get("min_id")) + end = params.get("end_id", params.get("max_id")) + event = params.get("id") + # `id` on block means block ID, so its caller does not pass it here. + for endpoint in (start, end, event): + if endpoint is not None and (type(endpoint) is not int or not low <= endpoint <= high): + raise InvalidParameterError("Event range must lie within the selected real scope.") + low = cast(int, start) if start is not None else low + high = cast(int, end) if end is not None else high + if event is not None: + low, high = max(low, cast(int, event)), min(high, cast(int, event)) + # Contradictory filters remain an empty matching set, not a made-up event. + elif low > high: + raise InvalidParameterError("Event range start must not exceed end.") + return low, high, [s for s in members if s.end_event_id >= low and s.start_event_id <= high] + + +def _scope( + sources: DatasetSourceResolver, low: int, high: int, items: list[SliceRecord] +) -> dict[str, object]: + return { + "kind": "event_range", + "device_id": sources.device.device_id, + "slice_index": items[0].index if len(items) == 1 else None, + "slice_indices": [s.index for s in items], + "first_event_id": low, + "last_event_id": high, + "requested_range": {"first_event_id": low, "last_event_id": high}, + "dataset_path": str(sources.dataset.root), + "fingerprint": sources.dataset.fingerprint, + "boundary_events_included": False, + "range_complete": True, + "work_budget": { + "max_rows": sources.budget.max_work_rows, + "max_bytes": sources.budget.max_work_bytes, + "unit": "cumulative_fetched_and_output_serialized_values; not_RSS", + }, + } + + +def _traces(sources: DatasetSourceResolver, low: int, high: int, items: list[SliceRecord]): + for item in items: + rows = sources.read( + sources.dataset.root / item.file, + f'SELECT * FROM "trace_entry_{sources.device.device_id}" WHERE id>=0 AND id>=? AND id<=? ORDER BY id', + [low, high], + ) + yield item, rows + + +def _peaks(sources, low, high, items, template, scope): + chosen: dict[str, dict[str, object]] = {} + count, first, last = 0, None, None + for _item, rows in _traces(sources, low, high, items): + for row in rows: + sources.budget.remaining() + event = cast(int, row["id"]) + count += 1 + first = event if first is None else min(first, event) + last = event if last is None else max(last, event) + for metric in METRICS: + if metric not in chosen or (cast(int, row[metric]), -event) > ( + cast(int, chosen[metric][metric]), + -cast(int, chosen[metric]["id"]), + ): + chosen[metric] = row + result: dict[str, object] = {} + for metric in METRICS: + row = chosen.get(metric, {}) + result[f"peak_{metric}"] = row.get(metric) + result[f"peak_{metric}_event_id"] = row.get("id") + if template == "allocator_gap": + for other in METRICS: + if other != metric: + result[f"{other}_at_{metric}_peak"] = row.get(other) + for other in ("active", "allocated"): + result[f"reserved_{other}_gap_at_{metric}_peak"] = ( + cast(int, row["reserved"]) - cast(int, row[other]) if row else None + ) + if template == "allocator_gap": + ids = [result[f"peak_{m}_event_id"] for m in METRICS] + result["all_peaks_same_event"] = int(None not in ids and len(set(ids)) == 1) + for a, b in (("allocated", "active"), ("active", "reserved"), ("allocated", "reserved")): + result[f"{a}_{b}_same_event"] = int( + result[f"peak_{a}_event_id"] is not None + and result[f"peak_{a}_event_id"] == result[f"peak_{b}_event_id"] + ) + scope["actual_range"] = { + "first_event_id": first, + "last_event_id": last, + "real_event_count": count, + } + scope["peak_owner_slices"] = { + m: sources.owner(cast(int, r["id"])).index for m, r in chosen.items() + } + return [result] + + +def _matches(row: dict[str, object], params: dict[str, object], fields: tuple[str, ...]) -> bool: + for field in fields: + value = params.get(field) + if value is not None and row[field] != value: + return False + for prefix, predicate in (("min_", lambda a, b: a >= b), ("max_", lambda a, b: a <= b)): + value = params.get(prefix + field) + if value is not None and (row[field] is None or not predicate(row[field], value)): + return False + return True + + +def _sort(rows: list[dict[str, object]], key: str, descending: bool) -> None: + # SQLite NULL placement and declared primary/id direction, plus full identity + # tie-breaks for unusual unproved historical rows with the same local ID. + rows.sort(key=lambda r: cast(str, r.get("lifecycle_id", "")), reverse=descending) + rows.sort(key=lambda r: cast(int, r["id"]), reverse=descending) + rows.sort(key=lambda r: (r[key] is not None, r[key]), reverse=descending) + + +def _events(sources, low, high, items, params, template, scope): + rows = [] + first, last, count = None, None, 0 + for _item, batch in _traces(sources, low, high, items): + for row in batch: + sources.budget.remaining() + event = cast(int, row["id"]) + first = event if first is None else min(first, event) + last = event if last is None else max(last, event) + count += 1 + fields = ("allocated", "active", "reserved") + if template == "event": + fields += ("id", "action", "address", "size", "stream") + if _matches(row, params, fields): + rows.append(row) + scope["actual_range"] = { + "first_event_id": first, + "last_event_id": last, + "real_event_count": count, + } + if template == "event": + # Only returned events need formatted text; resolve separately after the + # global window below. `order_by=callstack` requires text BEFORE sorting. + if params["order_by"] == "callstack": + refs = sources.events(cast(int, r["id"]) for r in rows) + for row in rows: + row["callstack"] = refs[cast(int, row["id"])].event.get("callstack") + _sort(rows, params["order_by"], params["order_dir"] == "DESC") + return rows + + +def _lifecycles(sources, items, scope): + merged: dict[str, dict[str, object]] = {} + terminal = sources.device.slices[-1].index + resolved_alloc = resolved_free = unproved = 0 + for item in items: + blocks = sources.read( + sources.dataset.root / item.file, + f'SELECT * FROM "block_{sources.device.device_id}" ORDER BY id', + ) + refs = sources.events( + cast(int, row[field]) + for row in blocks + for field in ("allocEventId", "freeEventId") + if type(row[field]) is int and cast(int, row[field]) >= 0 + ) + for block in blocks: + sources.budget.remaining() + alloc, free = cast(int, block["allocEventId"]), block["freeEventId"] + allocation = refs.get(alloc) + allocation = allocation if allocation and allocation.event["action"] == 4 else None + completion = refs.get(cast(int, free)) + completion = completion if completion and completion.event["action"] == 6 else None + proved = allocation is not None + stable_negative = alloc == -1 and cast(int, block["id"]) < 0 + token = ( + f"alloc:{alloc}" + if proved + else ( + f"preexisting:{block['id']}" + if stable_negative + else f"unproved:slice:{item.index}:block:{block['id']}" + ) + ) + identity = f"{sources.namespace}:{token}" + old = merged.get(identity) + if old is not None: + # A shared token with incompatible invariant observations is not + # evidence to merge by address or guess which allocation wins. + if any( + old[f] != block[f] for f in ("address", "size", "requestedSize", "allocEventId") + ): + raise QueryExecutionError( + "Conflicting observations for a proved lifecycle identity." + ) + if old["free_source"] and completion and old["freeEventId"] != free: + raise QueryExecutionError( + "Conflicting free-completion events for one lifecycle." + ) + if old["free_source"] and not completion: + free = old["freeEventId"] + completion_dict = old["free_source"] + else: + completion_dict = completion.to_dict() if completion else None + else: + completion_dict = completion.to_dict() if completion else None + block.update( + { + "lifecycle_id": identity, + "identity_status": ( + "proved_allocation_event" + if proved + else "stable_negative_token" if stable_negative else "unproved_shard_local" + ), + "allocation_source": allocation.to_dict() if allocation else None, + "allocation_source_status": ( + "resolved" + if proved + else "unknown_preexisting" if alloc == -1 else "missing" + ), + "free_source": completion_dict, + "free_source_status": ( + "resolved" + if completion_dict + else "live_or_unknown" if free is None or cast(int, free) < 0 else "missing" + ), + "freeEventId": free, + "observation_slice_index": item.index, + "state_scope": "latest_slice_observation", + "terminal_survivor": item.index == terminal + and block["state"] in (0, 1) + and (free is None or cast(int, free) < 0), + "category": ( + "dynamic" if alloc >= 0 else "static" if free == -1 else "preexisting" + ), + } + ) + merged[identity] = block + rows = list(merged.values()) + for row in rows: + resolved_alloc += row["allocation_source"] is not None + resolved_free += row["free_source"] is not None + unproved += row["identity_status"] == "unproved_shard_local" + scope["source_coverage"] = { + "coverage_scope": "all_deduplicated_lifecycles_before_filters_and_page", + "range_complete": True, + "lifecycles": len(rows), + "allocation_events_resolved": resolved_alloc, + "free_events_resolved": resolved_free, + "unproved_identities": unproved, + "allocation_source_complete": all( + r["allocation_source"] is not None for r in rows if cast(int, r["allocEventId"]) >= 0 + ), + "source_queries": sources.query_count, + } + return rows + + +def _stacks(sources, low, high, items, params, scope): + grouped = {} + events = ordered = 0 + for _item, batch in _traces(sources, low, high, items): + refs = sources.events(cast(int, r["id"]) for r in batch) + for source in refs.values(): + sources.budget.remaining() + text = source.event.get("callstack") + # Historical alloc_count is ALL stack-bearing real trace actions, + # including free/segment/workspace. Never claim allocation-only. + referenced = ( + sources.dataset.callstack_layout == "v2" and source.local_stack_id is not None + ) + if text is None and source.frames is None and not referenced: + continue + identity = source.stack_id + # Point attribution normalizes empty text into missing; statistics + # historically count the non-NULL empty string as its own group. + if text == "" and source.frames is None: + identity = f"{sources.namespace}:text:sha256:{hashlib.sha256(b'').hexdigest()}" + row = grouped.setdefault( + identity, + { + "callstack": text if text is not None else "[missing callstack]", + "stack_kind": source.stack_kind, + "text_kind": source.text_kind, + "alloc_count": 0, + "total_size": 0, + "max_size": 0, + "source_stack_id": identity, + "stack_event_id": source.event["id"], + "frames_status": source.frames_status, + }, + ) + row["alloc_count"] += 1 + row["total_size"] += source.event["size"] + row["max_size"] = max(row["max_size"], source.event["size"]) + if row["text_kind"] != "captured" and source.text_kind == "captured": + row["callstack"], row["stack_event_id"] = text, source.event["id"] + row["text_kind"] = "captured" + events += 1 + ordered += source.frames is not None + rows = [ + r + for r in grouped.values() + if r["alloc_count"] >= params["min_count"] and r["total_size"] >= params["min_size"] + ] + for row in rows: + row["avg_size"] = row["total_size"] / row["alloc_count"] + rows.sort( + key=lambda r: ( + -r["total_size"], + r["callstack"] or "", + r["stack_kind"] == "missing", + r["source_stack_id"], + ) + ) + scope["source_coverage"] = { + "range_complete": True, + "coverage_scope": "all_real_stack_bearing_actions_before_global_thresholds", + "stack_events": events, + "ordered_frame_events": ordered, + "source_queries": sources.query_count, + } + return rows + + +def global_query( + sources: DatasetSourceResolver, + template: str, + params: dict[str, object], + max_rows: int | None, + exact_total: bool, + slice_index: int | None, + semantics_version: int | None, +) -> QueryResult: + range_params = ( + params if template in ("event", "allocation", "memory_peak", "allocator_gap") else {} + ) + low, high, items = _range(sources, range_params, slice_index) + scope = _scope(sources, low, high, items) + if template in ("memory_peak", "allocator_gap"): + rows = _peaks(sources, low, high, items, template, scope) + elif template in ("event", "allocation"): + rows = _events(sources, low, high, items, params, template, scope) + elif template == "callstack_analysis": + rows = _stacks(sources, low, high, items, params, scope) + elif template == "preexisting_live": + event = cast(int, params["event_id"]) + item = sources.target(event, slice_index) + blocks = sources.read( + sources.dataset.root / item.file, + f'SELECT COUNT(*) AS block_count, COALESCE(SUM(size),0) AS size_bytes FROM "block_{sources.device.device_id}" WHERE allocEventId=-1 AND (freeEventId IS NULL OR freeEventId> ? OR (freeEventId<0 AND freeEventId!=-1))', + [event], + ) + rows = blocks + scope = _scope(sources, event, event, [item]) + else: + if template == "leak_detection" and slice_index is not None: + raise QueryExecutionError( + "Dataset leak_detection requires terminal dataset scope, not --slice; select an explicit member for local observations." + ) + rows = _lifecycles(sources, items, scope) + if template == "block": + rows = [ + r + for r in rows + if _matches( + r, + params, + ("id", "address", "size", "requestedSize", "allocEventId", "freeEventId"), + ) + ] + _sort(rows, cast(str, params["order_by"]), params["order_dir"] == "DESC") + elif template == "leak_detection": + rows = [ + r + for r in rows + if r["terminal_survivor"] + and r["allocation_source"] is not None + and cast(int, r["size"]) >= cast(int, params["min_size"]) + ] + _sort(rows, "size", True) + elif template == "freed_block_lifetime": + buckets = {} + for row in rows: + if row["allocation_source"] is None or row["free_source"] is None: + continue + distance = cast(int, row["freeEventId"]) - cast(int, row["allocEventId"]) + if distance < 0: + continue + index = next( + (i for i, cap in enumerate((1000, 5000, 20000, 100000)) if distance < cap), 4 + ) + bucket = buckets.setdefault( + index, + { + "lifetime_events": ("<1k", "1k-5k", "5k-20k", "20k-100k", ">=100k")[index], + "block_count": 0, + "size_bytes": 0, + }, + ) + bucket["block_count"] += 1 + bucket["size_bytes"] += row["size"] + rows = [buckets[i] for i in sorted(buckets)] + else: + raise QueryExecutionError("No dataset-global merge implementation for this template.") + total = len(rows) + offset = max(0, cast(int, params.get("offset", 0))) + limit = cast(int, params.get("limit", -1)) + if max_rows is not None and max_rows > 0: + limit = min(limit, max_rows) if limit >= 0 else max_rows + selected = rows[offset : offset + limit] if limit >= 0 else rows[offset:] + if template == "event": + refs = sources.events(cast(int, r["id"]) for r in selected) + detailed = ( + sources.dataset.ordered_frames_version == 1 or cast(int, params["stack_bytes"]) >= 0 + ) + selected = [dict(refs[cast(int, r["id"])].event) for r in selected] + if detailed: + for row in selected: + source = refs[cast(int, row["id"])] + row.update( + { + "source_stack_id": source.stack_id, + "stack_id": source.stack_id, + "stack_kind": source.stack_kind, + "text_kind": source.text_kind, + "stack_event_id": row["id"] if source.stack_kind == "captured" else None, + "frames": source.frames, + "frames_status": source.frames_status, + } + ) + scope["source_coverage"] = { + "range_complete": True, + "event_position": "after", + "coverage_scope": "returned_events", + "events_resolved": len(refs), + "ordered_frame_events": sum(s.frames is not None for s in refs.values()), + "source_queries": sources.query_count, + } + elif template == "allocation": + selected = [{k: r[k] for k in ("id", *METRICS)} for r in selected] + sources.budget.consume(selected) + more = total > offset + len(selected) + return QueryResult( + total=total if exact_total else len(selected), + returned=len(selected), + device_id=sources.device.device_id, + rows=selected, + template=template, + semantics_version=semantics_version, + has_more=more, + truncated=more or offset > 0, + total_is_exact=exact_total or not more and offset == 0, + timeout_s=sources.budget.timeout_s, + scope=scope, + ) diff --git a/src/pt_snap_cli/core/dataset_import_backend.py b/src/pt_snap_cli/core/dataset_import_backend.py new file mode 100644 index 0000000..ab786d6 --- /dev/null +++ b/src/pt_snap_cli/core/dataset_import_backend.py @@ -0,0 +1,219 @@ +"""Native dataset publication: exclusive renames, explicit compensation, no crash guarantee.""" + +from __future__ import annotations + +import shutil +import tempfile +from collections.abc import Callable +from dataclasses import dataclass +from pathlib import Path + +from .dataset_contract import DatasetContractError +from .dataset_resolver import _hash_file +from .errors import ImportExecutionError +from .native_dataset_contract import NativeManifest, read_native_manifest, require_native_members +from .sharded_replay_service import ShardedReplayResult, ShardedReplayService +from .split_service import SplitService + +DirectoryIdentity = tuple[int, int, int | None] + + +@dataclass(frozen=True) +class TargetState: + device: int + inode: int + files: tuple[tuple[str, str], ...] + + +class DatasetImportBackend: + def __init__(self, replay: ShardedReplayService | None = None): + self.replay = replay or ShardedReplayService() + + @staticmethod + def target_path(source: Path, output_dir: Path) -> Path: + return output_dir / f"{source.name}.pt-snap-native-v2" + + @staticmethod + def inspect_target(target: Path) -> tuple[NativeManifest, TargetState] | None: + if not target.exists() and not target.is_symlink(): + return None + try: + if target.resolve() != target or target.is_symlink() or not target.is_dir(): + raise ValueError("Target must be a canonical non-symlink native dataset directory") + manifest = read_native_manifest(target) + require_native_members(target, manifest) + status = target.lstat() + files = tuple( + (name, _hash_file(target / name)) + for name in sorted( + ["manifest.json", *(s.file for d in manifest.devices for s in d.slices)] + ) + ) + return manifest, TargetState(status.st_dev, status.st_ino, files) + except (DatasetContractError, OSError, ValueError) as exc: + raise ImportExecutionError( + f"Existing target is not a recognized native dataset; preserved: {exc}" + ) from exc + + @staticmethod + def _verify(path: Path, identity: DirectoryIdentity) -> None: + if path.resolve() != path: + raise OSError(f"Noncanonical or substituted directory: {path}") + SplitService._verify_stage_identity(path, identity) + + @classmethod + def _remove(cls, path: Path, identity: DirectoryIdentity) -> None: + cls._verify(path, identity) + # shutil.rmtree does not traverse child symlink targets. The root is held + # by an inode-checked descriptor from secure task allocation/recognition. + shutil.rmtree(path) + + @classmethod + def _move(cls, source: Path, destination: Path, identity: DirectoryIdentity) -> None: + cls._verify(source, identity) + if destination.parent.resolve() != destination.parent or destination.is_symlink(): + raise OSError(f"Unsafe publication destination: {destination}") + SplitService._publish_directory(source, destination) + + @classmethod + def _matches(cls, path: Path, identity: DirectoryIdentity) -> bool: + try: + cls._verify(path, identity) + return True + except OSError: + return False + + def dump_to_dataset( + self, + source: Path, + target: Path, + *, + capacity: int, + device: int | None, + expected: TargetState | None, + finalize: Callable[[ShardedReplayResult], None], + post_publish: Callable[[Path], None] | None = None, + pre_publish: Callable[[], None] | None = None, + ) -> Path: + if ( + target.resolve() != target + or target.parent.resolve() != target.parent + or target.is_symlink() + ): + raise ImportExecutionError("Dataset output must be canonical and non-symlink.") + stage: Path | None = None + recovery: Path | None = None + stage_id: DirectoryIdentity | None = None + recovery_id: DirectoryIdentity | None = None + old_id: DirectoryIdentity | None = None + new_id: DirectoryIdentity | None = None + preserve = False + try: + SplitService._preflight_publication() + target.parent.mkdir(parents=True, exist_ok=True) + current = self.inspect_target(target) + if (current[1] if current else None) != expected: + raise OSError("Dataset destination changed before staging; preserved") + if expected is not None: + old_id = SplitService._stage_identity(target) + stage = Path(tempfile.mkdtemp(dir=target.parent, prefix=f".{target.name}.stage-")) + stage_id = SplitService._stage_identity(stage) + artifact = stage / "artifact" + result = self.replay.stage(source, artifact, events_per_slice=capacity, device=device) + self._verify(stage, stage_id) + new_id = SplitService._stage_identity(artifact) + finalize(result) + self._verify(artifact, new_id) + current = self.inspect_target(target) if expected is not None else None + if (current[1] if current is not None else None) != expected: + raise OSError("Dataset destination changed during import; preserved") + if pre_publish is not None: + pre_publish() + try: + if old_id is not None: + self._verify(target, old_id) + recovery = Path( + tempfile.mkdtemp(dir=target.parent, prefix=f".{target.name}.recovery-") + ) + recovery_id = SplitService._stage_identity(recovery) + self._move(target, recovery / "previous", old_id) + self._move(artifact, target, new_id) + if post_publish is not None: + post_publish(target) + except BaseException as publish_error: + try: + # Detect successful rename BEFORE an injected call raises. + # Never remove an independently created/substituted target. + if self._matches(target, new_id): + self._remove(target, new_id) + if old_id is not None: + if not self._matches(target, old_id): + if ( + recovery is None + or recovery_id is None + or not self._matches(recovery / "previous", old_id) + ): + raise OSError( + "Previous artifact is missing or substituted; recovery cannot be trusted" + ) + self._verify(recovery, recovery_id) + previous = self.inspect_target(recovery / "previous") + if previous is None or previous[1] != expected: + raise OSError( + "Recovery artifact changed; preserve evidence instead of restoring unknown data" + ) + try: + self._move(recovery / "previous", target, old_id) + except BaseException: + # A late exception may follow a successful restore. + restored = self.inspect_target(target) + if ( + not self._matches(target, old_id) + or restored is None + or restored[1] != expected + ): + raise + restored = self.inspect_target(target) + if restored is None or restored[1] != expected: + raise OSError( + "Previous artifact was not restored exactly; preserve recovery evidence" + ) + except BaseException as rollback_error: + preserve = True + raise ImportExecutionError( + f"Dataset publication/focus failed; directory rollback also failed: {rollback_error}. " + f"Recovery directory: {recovery}; staging evidence: {stage}." + ) from publish_error + raise + # The transaction is committed only after the focus callback returns. + if recovery is not None and old_id is not None: + try: + previous = self.inspect_target(recovery / "previous") + if previous is None or previous[1] != expected: + raise OSError("Recovery artifact changed; do not delete unknown contents") + self._remove(recovery / "previous", old_id) + except Exception as exc: + preserve = True + raise ImportExecutionError( + f"Dataset publication committed but recovery cleanup failed; preserved evidence at {recovery}: {exc}" + ) from exc + return target + except ImportExecutionError: + raise + except Exception as exc: + raise ImportExecutionError(f"Dataset import backend failed: {exc}") from exc + finally: + try: + if not preserve: + for path, identity in ((stage, stage_id), (recovery, recovery_id)): + if path is not None and identity is not None: + try: + self._remove(path, identity) + except OSError as exc: + raise ImportExecutionError( + f"Dataset cleanup failed; preserved recovery evidence at {path}: {exc}" + ) from exc + finally: + for identity in (new_id, old_id, stage_id, recovery_id): + if identity is not None: + SplitService._close_stage_identity(identity) diff --git a/src/pt_snap_cli/core/dataset_resolver.py b/src/pt_snap_cli/core/dataset_resolver.py new file mode 100644 index 0000000..94126ec --- /dev/null +++ b/src/pt_snap_cli/core/dataset_resolver.py @@ -0,0 +1,177 @@ +"""Resolve finalized compatibility datasets above the single-DB Context layer. + +Original local reader of the P0 contract; no pickle, migration or producer code. +Native-v2 private replay staging is not a dataset until the native publisher closes, +attests and validates its complete versioned manifest. +""" + +from __future__ import annotations + +import hashlib +import json +from dataclasses import dataclass +from pathlib import Path + +from pt_snap_cli.core.dataset_contract import ( + DatasetContractError, + DatasetValidation, + DeviceRecord, + QueryScope, + _unique_object, + parse_manifest, + validate_dataset, +) +from pt_snap_cli.core.dataset_files import hash_file as _hash_file +from pt_snap_cli.core.dataset_files import require_readonly_member as _require_readonly_member +from pt_snap_cli.core.dataset_support import DATASET_SUPPORT +from pt_snap_cli.core.errors import DatabaseSchemaError, InvalidDeviceError, InvalidParameterError +from pt_snap_cli.core.native_dataset_contract import ( + NATIVE_FORMAT, + NativeManifest, + parse_native_manifest, + validate_native_dataset, + validate_native_scope, +) + + +@dataclass(frozen=True) +class ResolvedDataset: + root: Path + validation: DatasetValidation + fingerprint: str + ordered_frames_version: int | None = None + + @property + def device_ids(self) -> list[int]: + return sorted(item.device_id for item in self.validation.manifest.devices) + + @property + def callstack_layout(self) -> str: + return "v2" if isinstance(self.validation.manifest, NativeManifest) else "v1" + + @property + def callstack_layout_error(self) -> None: + return None + + def device(self, device_id: int | None = None) -> DeviceRecord: + selected = self.device_ids[0] if device_id is None else device_id + for device in self.validation.manifest.devices: + if type(selected) is int and device.device_id == selected: + return device + raise InvalidDeviceError(f"Device {selected} not found. Available: {self.device_ids}") + + def paths(self, scope: QueryScope) -> tuple[Path, ...]: + """Address all matching slices; this does not execute/aggregate queries.""" + try: + manifest = self.validation.manifest + if isinstance(manifest, NativeManifest): + validate_native_scope(scope, manifest) + else: + scope.validate(manifest) + except DatasetContractError as exc: + raise InvalidParameterError(str(exc)) from exc + return tuple( + self.root / item.file + for device in self.validation.manifest.devices + if scope.device_id is None or device.device_id == scope.device_id + for item in device.slices + if (scope.slice_index is None or item.index == scope.slice_index) + and (scope.start_event_id is None or item.end_event_id >= scope.start_event_id) + and (scope.end_event_id is None or item.start_event_id <= scope.end_event_id) + ) + + def to_dict(self) -> dict[str, object]: + manifest = self.validation.manifest + return { + "manifest_path": str(self.root / "manifest.json"), + "schema_version": manifest.schema_version, + "status": manifest.status, + "format": NATIVE_FORMAT if isinstance(manifest, NativeManifest) else "compatibility-v1", + "fingerprint": self.fingerprint, + "real_event_count": self.validation.real_event_count, + "boundary_event_count": self.validation.boundary_event_count, + "text_callstacks": True, + "structured_frames": False, + "cross_slice_queries": True, + "supported_queries": dict(DATASET_SUPPORT), + "cross_slice_sources": True, + "ordered_frames_reader_version": 1, + "structured_frames_scope": "validation_only; query source coverage is separate", + "devices": [ + { + "device_id": device.device_id, + "event_count": device.event_count, + "slices": [ + { + "index": item.index, + "first_event_id": item.start_event_id, + "last_event_id": item.end_event_id, + "db_path": str(self.root / item.file), + } + for item in device.slices + ], + } + for device in sorted(manifest.devices, key=lambda item: item.device_id) + ], + } + + +class DatasetResolver: + """Stateless resolution: revalidate every call, retain no connections/manifest cache. + + Content hashes (not cacheHash or size/mtime alone) form the Context generation. + Work is proportional to artifact size; no concurrent-publication lock is claimed. + """ + + def inspect(self, path: Path | str) -> ResolvedDataset | None: + candidate = Path(path).expanduser().absolute() + if not candidate.is_dir() and candidate.name != "manifest.json": + return None + root = candidate if candidate.is_dir() else candidate.parent + if candidate.is_symlink() or root.resolve() != root: + raise DatabaseSchemaError("Dataset must use canonical non-symlink paths.") + try: + manifest_path = root / "manifest.json" + if manifest_path.is_symlink(): + raise DatabaseSchemaError("Dataset manifest must not be a symlink.") + before = _hash_file(manifest_path) + with manifest_path.open(encoding="utf-8") as source: + raw: object = json.load(source, object_pairs_hook=_unique_object) + planned = ( + parse_native_manifest(raw) + if isinstance(raw, dict) and raw.get("format") == NATIVE_FORMAT + else parse_manifest(raw) + ) + # mode=ro can create WAL/SHM files even after the last writer has + # checkpointed and removed them. Check every member before the first + # SQLite open, without following aliases or repairing the artifact. + for device in planned.devices: + for item in device.slices: + _require_readonly_member( + root, root / item.file, immutable=not isinstance(planned, NativeManifest) + ) + validation = ( + validate_native_dataset(root, planned) + if isinstance(planned, NativeManifest) + else validate_dataset(root) + ) + digest = hashlib.sha256() + digest.update(before.encode("ascii")) + for device in validation.manifest.devices: + for item in device.slices: + member = root / item.file + _require_readonly_member( + root, member, immutable=not isinstance(planned, NativeManifest) + ) + digest.update(item.file.encode("utf-8")) + digest.update(_hash_file(member).encode("ascii")) + if before != _hash_file(manifest_path): + raise DatabaseSchemaError("Dataset manifest changed during inspection; retry.") + extensions = raw.get("extensions") if isinstance(raw, dict) else None + frames = extensions.get("ptSnapOrderedFrames") if isinstance(extensions, dict) else None + version = frames.get("version") if isinstance(frames, dict) else None + return ResolvedDataset( + root, validation, digest.hexdigest(), version if type(version) is int else None + ) + except (DatasetContractError, OSError, UnicodeError, json.JSONDecodeError) as exc: + raise DatabaseSchemaError(f"Invalid dataset: {exc}") from exc diff --git a/src/pt_snap_cli/core/dataset_sources.py b/src/pt_snap_cli/core/dataset_sources.py new file mode 100644 index 0000000..7a8e3f0 --- /dev/null +++ b/src/pt_snap_cli/core/dataset_sources.py @@ -0,0 +1,345 @@ +"""Read-only, batched device/event source lookup over a validated dataset. + +No address joins, foreign-event insertion, text-to-frame reconstruction, or retained +source cache. The caller owns the Context LRU and the entire operation's deadline. +""" + +from __future__ import annotations + +import hashlib +import json +import math +import sqlite3 +import time +from bisect import bisect_right +from collections import defaultdict +from collections.abc import Iterable +from dataclasses import dataclass +from pathlib import Path +from typing import cast + +from pt_snap_cli.context import DatabaseNotFoundError, SchemaVersionError +from pt_snap_cli.core.context_cache import ContextCache +from pt_snap_cli.core.dataset_contract import SliceRecord, _unique_object +from pt_snap_cli.core.dataset_files import require_readonly_member +from pt_snap_cli.core.dataset_resolver import ResolvedDataset +from pt_snap_cli.core.errors import ( + DatabaseSchemaError, + InvalidParameterError, + QueryExecutionError, + QueryTimeoutError, +) +from pt_snap_cli.query.executor import QueryExecutionError as ExecutorError +from pt_snap_cli.query.executor import QueryExecutor +from pt_snap_cli.query.executor import QueryTimeoutError as ExecutorTimeout + +SOURCE_BATCH_SIZE = 256 + + +def _invalid_constant(value: str) -> object: + raise ValueError(f"Non-JSON numeric constant: {value}") + + +def _finite_float(value: str) -> float: + result = float(value) + if not math.isfinite(result): + raise ValueError("Non-finite frame number") + return result + + +def _bounded_depth(value: object) -> bool: + pending = [(value, 0)] + while pending: + current, depth = pending.pop() + if depth > 128: + return False + if isinstance(current, dict): + pending.extend((item, depth + 1) for item in current.values()) + elif isinstance(current, list): + pending.extend((item, depth + 1) for item in current) + return True + + +@dataclass +class QueryBudget: + timeout_s: float | None + started: float + max_work_rows: int = 100_000 + max_work_bytes: int = 64 * 1024 * 1024 + work_rows: int = 0 + work_bytes: int = 0 + + def consume(self, rows: list[dict[str, object]]) -> None: + """Bound cumulative fetched rows/serialized values, not merely output. + + Includes repeated source reads. Exceeding either ceiling fails the whole + operation, never returning a silently partial global conclusion. This is + not an RSS ceiling: SQLite and one fetched batch have their own overhead. + """ + self.remaining() + self.work_rows += len(rows) + self.work_bytes += sum( + len(json.dumps(row, ensure_ascii=True, default=str).encode("utf-8")) for row in rows + ) + if self.work_rows > self.max_work_rows or self.work_bytes > self.max_work_bytes: + raise QueryExecutionError( + "Dataset query work budget exceeded (100000 fetched rows / 64 MiB serialized " + "values by default); no partial global result. Narrow the scope or select a member." + ) + + def remaining(self) -> float | None: + if self.timeout_s is None: + return None + left = self.timeout_s - (time.monotonic() - self.started) + if left <= 0: + raise QueryTimeoutError(f"Query timed out after {self.timeout_s} seconds") + return left + + +@dataclass(frozen=True) +class EventSource: + event: dict[str, object] + slice_index: int + stack_id: str + stack_kind: str + frames: list[dict[str, object]] | None = None + frames_status: str = "text_only" + local_stack_id: int | None = None + + @property + def text_kind(self) -> str: + text = self.event.get("callstack") + return "captured" if isinstance(text, str) and text != "" else "missing" + + def to_dict(self) -> dict[str, object]: + return { + "event": self.event, + "slice_index": self.slice_index, + "local_stack_id": self.local_stack_id, + "stack_id": self.stack_id, + "stack_kind": self.stack_kind, + "text_kind": self.text_kind, + "frames": self.frames, + "frames_status": self.frames_status, + } + + +class DatasetSourceResolver: + """One operation's sources; at most one SQL batch per shard/256 distinct IDs. + + Schema/optional frame checks add a constant number of queries per visited + shard. Connections are sequential and bounded by the borrowed LRU, not blocks. + """ + + def __init__( + self, dataset: ResolvedDataset, device_id: int, cache: ContextCache, budget: QueryBudget + ) -> None: + self.dataset = dataset + self.device = dataset.device(device_id) + self.cache = cache + self.budget = budget + self.query_count = 0 + self._starts = [s.start_event_id for s in self.device.slices] + + @property + def namespace(self) -> str: + return f"dataset:{self.dataset.fingerprint}:device:{self.device.device_id}" + + def owner(self, event_id: int) -> SliceRecord | None: + if type(event_id) is not int or event_id < 0: + return None + index = bisect_right(self._starts, event_id) - 1 + if index < 0: + return None + item = self.device.slices[index] + return item if event_id <= item.end_event_id else None + + def target(self, event_id: int, slice_index: int | None = None) -> SliceRecord: + item = self.owner(event_id) + if item is None or (slice_index is not None and item.index != slice_index): + raise InvalidParameterError("Event ID must be a real ID within the selected scope.") + if event_id not in self.events([event_id]): + raise InvalidParameterError( + f"No real event at ID {event_id}; sparse gaps are not events." + ) + return item + + def _consume(self, rows: list[dict[str, object]]) -> None: + # The executor intentionally knows only its own errors. Translate the + # core callback's timeout before its generic exception boundary, so + # cancellation keeps the domain kind and CLI QUERY_TIMEOUT code. + try: + self.budget.consume(rows) + except QueryTimeoutError as exc: + raise ExecutorTimeout(str(exc)) from exc + except QueryExecutionError as exc: + raise ExecutorError(str(exc)) from exc + + def read( + self, path: Path, sql: str, values: list[object] | None = None + ) -> list[dict[str, object]]: + self.budget.remaining() + require_readonly_member( + self.dataset.root, path, immutable=self.dataset.callstack_layout == "v1" + ) + try: + context = self.cache.get( + path, + generation=self.dataset.fingerprint, + immutable=self.dataset.callstack_layout == "v1", + ) + self.query_count += 1 + return QueryExecutor(context).execute( + sql, values, timeout_s=self.budget.remaining(), consume=self._consume + ) + except ExecutorTimeout as exc: + raise QueryTimeoutError( + f"Query timed out after {self.budget.timeout_s} seconds" + ) from exc + except ExecutorError as exc: + raise QueryExecutionError(str(exc)) from exc + except (DatabaseNotFoundError, SchemaVersionError, sqlite3.DatabaseError) as exc: + raise DatabaseSchemaError(str(exc)) from exc + + def events(self, event_ids: Iterable[int]) -> dict[int, EventSource]: + grouped: dict[int, list[int]] = defaultdict(list) + for event in sorted(set(event_ids)): + self.budget.remaining() + item = self.owner(event) + if item is not None: + grouped[item.index].append(event) + result: dict[int, EventSource] = {} + for index, ids in grouped.items(): + item = self.device.slices[index] + path = self.dataset.root / item.file + trace = f'"trace_entry_{self.device.device_id}"' + for start in range(0, len(ids), SOURCE_BATCH_SIZE): + batch = ids[start : start + SOURCE_BATCH_SIZE] + placeholders = ",".join("?" for _ in batch) + sql = ( + f"SELECT t.*, cs.callstack FROM {trace} t LEFT JOIN callstack cs ON t.callstackId=cs.id" + if self.dataset.callstack_layout == "v2" + else f"SELECT t.* FROM {trace} t" + ) + rows = self.read( + path, sql + f" WHERE t.id IN ({placeholders}) AND t.id>=0", list(batch) + ) + for row in rows: + event_id = cast(int, row["id"]) + text = row.get("callstack") + captured = isinstance(text, str) and text != "" + local_id = cast(int | None, row.pop("callstackId", None)) + # V2 IDs select text WITHIN the owning DB; they are provenance, + # not dataset grouping keys. Global interning equivalence uses + # the full captured text, never a shortened display prefix. + identity = ( + "text:sha256:" + + hashlib.sha256( + (text if isinstance(text, str) else "").encode("utf-8") + ).hexdigest() + ) + result[event_id] = EventSource( + row, + index, + ( + f"{self.namespace}:{identity}" + if captured + else f"{self.namespace}:category:missing" + ), + "captured" if captured else "missing", + local_stack_id=local_id, + ) + self._attach_frames(path, ids, result) + self.budget.remaining() + return result + + def _attach_frames(self, path: Path, ids: list[int], sources: dict[int, EventSource]) -> None: + # This is a NEW, explicit read contract, not a guessed original-producer + # schema. Existing pt_snap_block_reference has formatted text only. + if self.dataset.ordered_frames_version != 1: + return + expected = { + "pt_snap_frame_coverage": [("eventId", "INTEGER", 1), ("frameCount", "INTEGER", 0)], + "pt_snap_frame": [ + ("eventId", "INTEGER", 1), + ("frameIndex", "INTEGER", 2), + ("frameJson", "TEXT", 0), + ], + } + for table, columns in expected.items(): + kind = self.read(path, "SELECT type FROM sqlite_master WHERE name=?", [table]) + schema = self.read(path, f'PRAGMA table_info("{table}")') + if ( + kind != [{"type": "table"}] + or [(r["name"], r["type"], r["pk"]) for r in schema] != columns + ): + return # Unknown/absent schemas never imply precision. + for start in range(0, len(ids), SOURCE_BATCH_SIZE): + batch = ids[start : start + SOURCE_BATCH_SIZE] + placeholders = ",".join("?" for _ in batch) + coverage = self.read( + path, + f"SELECT * FROM pt_snap_frame_coverage WHERE eventId IN ({placeholders})", + list(batch), + ) + frames = self.read( + path, + f"SELECT * FROM pt_snap_frame WHERE eventId IN ({placeholders}) ORDER BY eventId,frameIndex", + list(batch), + ) + by_event: dict[int, list[dict[str, object]]] = defaultdict(list) + for row in frames: + by_event[cast(int, row["eventId"])].append(row) + for row in coverage: + event = cast(int, row["eventId"]) + count = row["frameCount"] + entries = by_event[event] + if ( + event not in sources + or type(count) is not int + or count < 0 + or len(entries) != count + ): + continue + if any( + type(entry["frameIndex"]) is not int or not isinstance(entry["frameJson"], str) + for entry in entries + ) or [entry["frameIndex"] for entry in entries] != list(range(count)): + continue + try: + decoded = [ + json.loads( + cast(str, entry["frameJson"]), + object_pairs_hook=_unique_object, + parse_constant=_invalid_constant, + parse_float=_finite_float, + ) + for entry in entries + ] + if not all(isinstance(frame, dict) for frame in decoded) or not _bounded_depth( + decoded + ): + continue + # Canonical raw arrays retain order, duplicates and typed fields. + # Text formatting and shard-local IDs are only provenance here. + digest = hashlib.sha256( + json.dumps( + decoded, + sort_keys=True, + ensure_ascii=True, + separators=(",", ":"), + allow_nan=False, + ).encode("utf-8") + ).hexdigest() + except (ValueError, TypeError, RecursionError, OverflowError): + continue + old = sources[event] + sources[event] = EventSource( + old.event, + old.slice_index, + f"{self.namespace}:frames:sha256:{digest}", + "captured", + decoded, + "ordered", + old.local_stack_id, + ) diff --git a/src/pt_snap_cli/core/dataset_support.py b/src/pt_snap_cli/core/dataset_support.py new file mode 100644 index 0000000..6d268dc --- /dev/null +++ b/src/pt_snap_cli/core/dataset_support.py @@ -0,0 +1,28 @@ +"""One shared, explicit per-template dataset execution support matrix.""" + +DATASET_SUPPORT: dict[str, str] = { + "event": "real_events_global_filter_sort_page", + "allocation": "real_events_global_filter_sort_page", + "memory_peak": "independent_real_counter_peaks_earliest_id", + "allocator_gap": "global_peaks_same_event_counters", + "active_blocks_at_event": "owner_event_active_set_cross_slice_sources", + "active_memory_callstack_at_event": "owner_event_global_source_groups_before_top_n", + "preexisting_live": "owner_event_preexisting_occupancy", + "block": "proved_lifecycles_dedup_latest_observation", + "freed_block_lifetime": "proved_completed_lifecycles_once", + "leak_detection": "terminal_dataset_dynamic_survivors_not_confirmed_leaks", + "callstack_analysis": "all_real_stack_bearing_actions_global_identity_weighted_stats", +} + + +def dataset_support(name: str, packaged: bool) -> dict[str, object]: + semantics = DATASET_SUPPORT.get(name) if packaged else None + return { + "supported": semantics is not None, + "semantics": semantics, + "device_scope": "one_selected_device; never sum device peaks", + "custom_sql": "unsupported; select an explicit standalone member", + "work_limit_rows": 100_000, + "work_limit_bytes": 64 * 1024 * 1024, + "budget_scope": "one_query_or_explicit_report_composition", + } diff --git a/src/pt_snap_cli/core/focus_service.py b/src/pt_snap_cli/core/focus_service.py index 45e31df..23e0b0b 100644 --- a/src/pt_snap_cli/core/focus_service.py +++ b/src/pt_snap_cli/core/focus_service.py @@ -1,11 +1,17 @@ from __future__ import annotations +import copy +import os import sqlite3 +import tempfile +from collections.abc import Iterator +from contextlib import contextmanager from pathlib import Path from typing import Any from pt_snap_cli.config import Config, FocusResolutionError from pt_snap_cli.context import Context, DatabaseNotFoundError, SchemaVersionError +from pt_snap_cli.core.dataset_resolver import DatasetResolver, ResolvedDataset from pt_snap_cli.core.errors import ( DatabaseMissingError, DatabaseSchemaError, @@ -55,7 +61,8 @@ def get_focus( if resolved.db_path is not None and resolved.db_path.exists(): try: - ctx = Context(resolved.db_path) + dataset = DatasetResolver().inspect(resolved.db_path) + ctx = dataset if dataset is not None else Context(resolved.db_path) available_devices = ctx.device_ids callstack_layout = ctx.callstack_layout callstack_layout_error = ctx.callstack_layout_error @@ -72,14 +79,130 @@ def get_focus( callstack_layout_error=callstack_layout_error, ) + @staticmethod + def _safe_focus_path(path: Path) -> None: + if path.resolve() != path or path.is_symlink() or (path.exists() and not path.is_file()): + raise OSError(f"Unsafe project focus path: {path}") + + @staticmethod + def _checkpoint_file(path: Path, content: bytes, mode: int) -> tuple[Path, tuple[int, int]]: + fd, name = tempfile.mkstemp(dir=path.parent, prefix=".focus.pt-snap-", suffix=".recovery") + backup = Path(name) + identity = os.fstat(fd) + try: + remaining = memoryview(content) + while remaining: + written = os.write(fd, remaining) + if written <= 0: + raise OSError("Could not write focus recovery checkpoint") + remaining = remaining[written:] + os.fchmod(fd, mode) + os.fsync(fd) + FocusService._verify_checkpoint(backup, (identity.st_dev, identity.st_ino)) + except BaseException: + try: + FocusService._verify_checkpoint(backup, (identity.st_dev, identity.st_ino)) + backup.unlink() + except OSError: + pass + raise + finally: + os.close(fd) + return backup, (identity.st_dev, identity.st_ino) + + @staticmethod + def _verify_checkpoint(path: Path, identity: tuple[int, int]) -> None: + FocusService._safe_focus_path(path) + info = path.lstat() + if (info.st_dev, info.st_ino) != identity: + raise OSError(f"Focus recovery checkpoint substituted: {path}") + + @staticmethod + def _restore_project_focus( + path: Path, backup: Path, identity: tuple[int, int], expected_bytes: bytes + ) -> None: + FocusService._safe_focus_path(path) + FocusService._verify_checkpoint(backup, identity) + if backup.read_bytes() != expected_bytes: + raise OSError("Focus recovery checkpoint content changed; preserve evidence") + os.replace(backup, path) + + @contextmanager + def project_focus_transaction(self) -> Iterator[None]: + """Import-only compensation, including writes that mutate and THEN raise. + + Store old bytes verbatim, not normalized JSON. An I/O/alias failure during + compensation preserves the checkpoint and reports its recovery location. + This is not a concurrency lock or an arbitrary-crash atomicity promise. + """ + from .errors import ImportExecutionError + + path = self._config.project_focus_path() + self._safe_focus_path(path) + old = path.read_bytes() if path.exists() else None + state = copy.deepcopy(self._config._config) + backup: Path | None = None + identity: tuple[int, int] | None = None + path.parent.mkdir(parents=True, exist_ok=True) + backup, identity = self._checkpoint_file( + path, + old if old is not None else b"", + path.stat().st_mode & 0o777 if old is not None else 0o600, + ) + preserve = False + try: + yield + except BaseException as focus_error: + self._config._config = state + try: + self._safe_focus_path(path) + if old is None: + if path.exists(): + # Verified exact path; never unlink a symlink target. + path.unlink() + elif not path.exists() or path.read_bytes() != old: + assert backup is not None and identity is not None + try: + self._restore_project_focus(path, backup, identity, old) + except BaseException: + # A rename may have restored the exact bytes BEFORE an + # injected/late I/O exception. Verify the actual outcome. + self._safe_focus_path(path) + if not path.exists() or path.read_bytes() != old: + raise + self._safe_focus_path(path) + if not path.exists() or path.read_bytes() != old: + raise OSError( + "Project focus was not restored exactly; preserve recovery evidence" + ) + if not backup.exists(): + backup = None + except BaseException as rollback_error: + preserve = True + raise ImportExecutionError( + f"Focus update failed and focus rollback also failed: {rollback_error}. " + f"Recovery focus checkpoint: {backup}; prior focus existed: {old is not None}." + ) from focus_error + raise + finally: + if backup is not None and identity is not None and not preserve: + # Failure to clean a committed checkpoint is safer than masking + # the real transaction outcome or deleting a substituted path. + try: + self._verify_checkpoint(backup, identity) + backup.unlink() + except OSError: + pass + def set_project_focus( self, db_path: Path | str, device_id: int | None = None, base_dir: Path | None = None, ) -> FocusState: - db_path = Path(db_path).expanduser().resolve() + db_path = Path(db_path).expanduser().absolute() ctx = self._validated_context(db_path) + db_path = db_path.resolve() self._validate_device(ctx, device_id) self._config.write_project_focus(db_path, base_dir=base_dir, device_id=device_id) return FocusState( @@ -93,8 +216,9 @@ def set_project_focus( ) def set_global_focus(self, db_path: Path | str, device_id: int | None = None) -> FocusState: - db_path = Path(db_path).expanduser().resolve() + db_path = Path(db_path).expanduser().absolute() ctx = self._validated_context(db_path) + db_path = db_path.resolve() self._validate_device(ctx, device_id) self._config.write_global_focus(db_path, device_id=device_id) return FocusState( @@ -161,8 +285,9 @@ def validate_session_db( db_path: Path | str, device_id: int | None = None, ) -> FocusState: - db_path = Path(db_path).expanduser().resolve() + db_path = Path(db_path).expanduser().absolute() ctx = self._validated_context(db_path) + db_path = db_path.resolve() self._validate_device(ctx, device_id) return FocusState( db_path=db_path, @@ -174,7 +299,10 @@ def validate_session_db( callstack_layout_error=ctx.callstack_layout_error, ) - def _validated_context(self, db_path: Path) -> Context: + def _validated_context(self, db_path: Path) -> Context | ResolvedDataset: + dataset = DatasetResolver().inspect(db_path) + if dataset is not None: + return dataset try: return Context(db_path) except DatabaseNotFoundError as exc: @@ -182,7 +310,7 @@ def _validated_context(self, db_path: Path) -> Context: except (SchemaVersionError, sqlite3.DatabaseError) as exc: raise DatabaseSchemaError(str(exc)) from exc - def _validate_device(self, ctx: Context, device_id: int | None) -> None: + def _validate_device(self, ctx: Context | ResolvedDataset, device_id: int | None) -> None: if device_id is None: return if device_id not in ctx.device_ids: diff --git a/src/pt_snap_cli/core/import_metadata.py b/src/pt_snap_cli/core/import_metadata.py index b2f9922..725c4f0 100644 --- a/src/pt_snap_cli/core/import_metadata.py +++ b/src/pt_snap_cli/core/import_metadata.py @@ -1,22 +1,29 @@ from __future__ import annotations import hashlib -import re +import json import sqlite3 from contextlib import closing from dataclasses import asdict -from datetime import datetime, timedelta, timezone +from datetime import datetime, timezone from pathlib import Path from typing import Any from pt_snap_cli import __version__ from pt_snap_cli.context import Context, DatabaseNotFoundError, SchemaVersionError +from pt_snap_cli.core.dataset_resolver import DatasetResolver from pt_snap_cli.core.errors import DatabaseMissingError, DatabaseSchemaError, ImportMetadataError +from pt_snap_cli.core.import_metadata_contract import ( + metadata_from_mapping, + validate_import_metadata, +) from pt_snap_cli.core.models import ( CacheDecision, ImportMetadata, MetadataInspection, ) +from pt_snap_cli.core.msinsight_export import inspect_owned_cache +from pt_snap_cli.core.native_dataset_contract import NativeManifest METADATA_TABLE = "pt_snap_metadata" METADATA_SCHEMA_VERSION = 1 @@ -26,7 +33,6 @@ IMPORTER_NAME = "pt-snap-cli" HASH_CHUNK_SIZE = 1024 * 1024 -_SHA256_RE = re.compile(r"^[0-9a-f]{64}$") _REQUIRED_COLUMNS = { "id", "metadata_schema_version", @@ -76,7 +82,22 @@ def build_metadata( ) def inspect(self, db_path: Path | str) -> MetadataInspection: + dataset = DatasetResolver().inspect(db_path) path = Path(db_path).expanduser().resolve() + if dataset is not None: + if isinstance(dataset.validation.manifest, NativeManifest): + return MetadataInspection(path, "available", dataset.validation.manifest.metadata) + # External P0 metadata remains optional. Only a recognized complete + # pt-snap manifest attestation can provide dataset-wide metadata. + with (dataset.root / "manifest.json").open(encoding="utf-8") as source: + raw = json.load(source) + if "ptSnap" in raw: + try: + owned = inspect_owned_cache(dataset.root) + except (ValueError, KeyError, TypeError, OSError, sqlite3.DatabaseError): + return MetadataInspection(path, "invalid", reason="metadata_invalid") + return MetadataInspection(path, "available", owned.metadata) + return MetadataInspection(path, "unavailable", reason="metadata_missing") try: context = Context(path) except DatabaseNotFoundError as exc: @@ -231,19 +252,7 @@ def inspection_to_dict(inspection: MetadataInspection) -> dict[str, Any]: } def _metadata_from_row(self, row: dict[str, Any]) -> ImportMetadata: - metadata = ImportMetadata( - metadata_schema_version=self._required_int(row["metadata_schema_version"]), - import_format_version=self._required_int(row["import_format_version"]), - source_sha256=self._required_str(row["source_sha256"]), - source_size=self._required_int(row["source_size"]), - source_name=self._required_str(row["source_name"]), - requested_device=self._optional_int(row["requested_device"]), - importer_name=self._required_str(row["importer_name"]), - importer_version=self._required_str(row["importer_version"]), - completed_at=self._required_str(row["completed_at"]), - ) - self._validate_metadata(metadata, allow_unsupported_version=True) - return metadata + return metadata_from_mapping(row) @staticmethod def _validate_metadata( @@ -256,19 +265,7 @@ def _validate_metadata( and metadata.metadata_schema_version != METADATA_SCHEMA_VERSION ): raise ValueError("Unsupported metadata schema version") - if metadata.metadata_schema_version < 1 or metadata.import_format_version < 1: - raise ValueError("Metadata versions must be positive integers") - if not _SHA256_RE.fullmatch(metadata.source_sha256): - raise ValueError("source_sha256 must be a lowercase SHA-256 digest") - if metadata.source_size < 0 or not metadata.source_name: - raise ValueError("Source metadata is invalid") - if metadata.requested_device is not None and metadata.requested_device < 0: - raise ValueError("requested_device must be non-negative") - if not metadata.importer_name or not metadata.importer_version: - raise ValueError("Importer metadata is invalid") - parsed_time = datetime.fromisoformat(metadata.completed_at) - if parsed_time.tzinfo is None or parsed_time.utcoffset() != timedelta(0): - raise ValueError("completed_at must use UTC") + validate_import_metadata(metadata) @staticmethod def _required_int(value: Any) -> int: diff --git a/src/pt_snap_cli/core/import_metadata_contract.py b/src/pt_snap_cli/core/import_metadata_contract.py new file mode 100644 index 0000000..b63cb73 --- /dev/null +++ b/src/pt_snap_cli/core/import_metadata_contract.py @@ -0,0 +1,55 @@ +"""Pure import-metadata value contract shared by standalone and dataset readers.""" + +from __future__ import annotations + +import re +from collections.abc import Mapping +from datetime import datetime, timedelta +from typing import cast + +from .models import ImportMetadata + + +def required_int(value: object) -> int: + if type(value) is not int: + raise TypeError("Expected integer metadata value") + return cast(int, value) + + +def required_str(value: object) -> str: + if not isinstance(value, str) or not value: + raise TypeError("Expected non-empty metadata string") + return value + + +def validate_import_metadata(metadata: ImportMetadata) -> None: + if metadata.metadata_schema_version < 1 or metadata.import_format_version < 1: + raise ValueError("Metadata versions must be positive integers") + if not re.fullmatch(r"[0-9a-f]{64}", metadata.source_sha256): + raise ValueError("source_sha256 must be a lowercase SHA-256 digest") + if metadata.source_size < 0 or not metadata.source_name: + raise ValueError("Source metadata is invalid") + if metadata.requested_device is not None and metadata.requested_device < 0: + raise ValueError("requested_device must be non-negative") + if not metadata.importer_name or not metadata.importer_version: + raise ValueError("Importer metadata is invalid") + parsed_time = datetime.fromisoformat(metadata.completed_at) + if parsed_time.tzinfo is None or parsed_time.utcoffset() != timedelta(0): + raise ValueError("completed_at must use UTC") + + +def metadata_from_mapping(row: Mapping[str, object]) -> ImportMetadata: + device = row["requested_device"] + metadata = ImportMetadata( + required_int(row["metadata_schema_version"]), + required_int(row["import_format_version"]), + required_str(row["source_sha256"]), + required_int(row["source_size"]), + required_str(row["source_name"]), + None if device is None else required_int(device), + required_str(row["importer_name"]), + required_str(row["importer_version"]), + required_str(row["completed_at"]), + ) + validate_import_metadata(metadata) + return metadata diff --git a/src/pt_snap_cli/core/import_service.py b/src/pt_snap_cli/core/import_service.py index 57c3600..7e8428f 100644 --- a/src/pt_snap_cli/core/import_service.py +++ b/src/pt_snap_cli/core/import_service.py @@ -1,20 +1,49 @@ from __future__ import annotations +import json import logging +import sqlite3 +from contextlib import nullcontext +from dataclasses import replace from pathlib import Path +from pt_snap_cli.core.dataset_import_backend import DatasetImportBackend +from pt_snap_cli.core.dataset_resolver import DatasetResolver, _require_readonly_member from pt_snap_cli.core.errors import ( DatabaseSchemaError, FocusFileInvalidError, ImportExecutionError, ImportMetadataError, InvalidDeviceError, + InvalidParameterError, SnapshotFileInvalidError, SourceChangedError, ) from pt_snap_cli.core.focus_service import FocusService from pt_snap_cli.core.import_metadata import ImportMetadataService -from pt_snap_cli.core.models import FocusState, ImportMetadata, ImportOptions, ImportResult +from pt_snap_cli.core.models import ( + CacheMissReason, + FocusState, + ImportMetadata, + ImportOptions, + ImportResult, +) +from pt_snap_cli.core.msinsight_export import ( + COMPATIBILITY_FORMAT, + DEFAULT_CAPACITY, + MsinsightImportBackend, + build_compatible_manifest, + compatible_identity, + convert_shards, + inspect_owned_cache, + salted_hash, +) +from pt_snap_cli.core.native_dataset_contract import ( + NATIVE_FORMAT, + build_native_manifest, + cache_identity, +) +from pt_snap_cli.core.sharded_replay_service import ShardedReplayResult from pt_snap_cli.core.snapshot_import_backend import SnapshotImportBackend logger = logging.getLogger(__name__) @@ -30,13 +59,21 @@ def __init__( focus_service: FocusService | None = None, backend: SnapshotImportBackend | None = None, metadata_service: ImportMetadataService | None = None, + dataset_backend: DatasetImportBackend | None = None, ) -> None: self._focus_service: FocusService | None = focus_service self._backend: SnapshotImportBackend = backend or SnapshotImportBackend() self._metadata_service: ImportMetadataService = metadata_service or ImportMetadataService() + self._dataset_backend = dataset_backend or DatasetImportBackend() + self._msinsight_backend = MsinsightImportBackend() def import_snapshot(self, options: ImportOptions) -> ImportResult: + self._validate_options(options) self._validate_snapshot_file(options.snapshot_file) + if options.format in (COMPATIBILITY_FORMAT, "msinsight"): + return self._import_compatible_dataset(options) + if options.events_per_slice is not None: + return self._import_dataset(options) output_dir = self._resolve_output_dir(options.snapshot_file, options.output_dir) db_path = self._backend.target_db_path(options.snapshot_file, output_dir) @@ -117,6 +154,246 @@ def set_focus_after_publish(published_db_path: Path) -> None: cache_miss_reason=decision.reason, ) + @staticmethod + def _validate_options(options: ImportOptions) -> None: + capacity = options.events_per_slice + if capacity is not None and (type(capacity) is not int or capacity <= 0): + raise InvalidParameterError("events_per_slice must be a positive integer.") + if options.device is not None and (type(options.device) is not int or options.device < 0): + raise InvalidParameterError("device must be a nonnegative integer.") + selected_format = ( + options.format + if options.format is not None + else (NATIVE_FORMAT if capacity is not None else "single-db") + ) + if selected_format in (COMPATIBILITY_FORMAT, "msinsight"): + return + if selected_format not in ("single-db", NATIVE_FORMAT): + raise InvalidParameterError(f"Unsupported import format: {selected_format}") + if (capacity is not None) != (selected_format == NATIVE_FORMAT): + raise InvalidParameterError( + "pt-snap-native-v2 requires --events-per-slice; single-db forbids it." + ) + + def _import_dataset(self, options: ImportOptions) -> ImportResult: + assert options.events_per_slice is not None + capacity = options.events_per_slice + output = ( + (options.output_dir if options.output_dir is not None else options.snapshot_file.parent) + .expanduser() + .absolute() + ) + target = self._dataset_backend.target_path(options.snapshot_file, output) + if target.resolve() != target or target.is_symlink(): + raise ImportExecutionError("Dataset output must be canonical and non-symlink.") + source_hash = self._hash_source(options.snapshot_file) + existing = self._dataset_backend.inspect_target(target) + reason: CacheMissReason = "database_missing" + if existing is not None: + manifest, expected = existing + reason = "forced" if options.force else "dataset_identity_changed" + try: + dataset = DatasetResolver().inspect(target) + except (DatabaseSchemaError, OSError): + dataset = None + if not options.force: + reason = "database_invalid" + if ( + not options.force + and dataset is not None + and manifest.identity == cache_identity(source_hash, options.device, capacity) + ): + self._recheck_source(options.snapshot_file, source_hash) + focus_state = self._set_focus_if_requested(target, options) + return ImportResult( + target, + options.device, + focus_state, + True, + manifest.metadata, + None, + target, + tuple(dataset.device_ids), + sum(len(d.slices) for d in manifest.devices), + NATIVE_FORMAT, + manifest.omitted_devices, + ) + if not options.force: + raise ImportExecutionError( + f"Existing dataset is not reusable ({reason}); use --force to replace this recognized native artifact. Existing target and focus preserved." + ) + else: + expected = None + completed: ImportMetadata | None = None + generated: ShardedReplayResult | None = None + + def finalize(result: ShardedReplayResult) -> None: + nonlocal completed, generated + metadata = self._metadata_service.build_metadata( + options.snapshot_file, source_hash, options.device + ) + for item in result.slices: + _require_readonly_member(result.directory, item.file) + self._metadata_service.write(item.file, metadata) + manifest = build_native_manifest(result, metadata, capacity) + with (result.directory / "manifest.json").open("x", encoding="utf-8") as file: + json.dump(manifest, file, indent=2) + validated = DatasetResolver().inspect(result.directory) + if validated is None: + raise ImportExecutionError( + "Staged dataset validation did not produce a complete artifact." + ) + completed, generated = metadata, result + + focus_state: FocusState | None = None + + def focus(path: Path) -> None: + nonlocal focus_state + focus_state = self._set_focus_if_requested(path, options) + + self._dataset_backend.dump_to_dataset( + options.snapshot_file, + target, + capacity=capacity, + device=options.device, + expected=expected, + finalize=finalize, + post_publish=focus if options.set_focus else None, + pre_publish=lambda: self._recheck_source(options.snapshot_file, source_hash), + ) + assert completed is not None and generated is not None + return ImportResult( + target, + options.device, + focus_state, + False, + completed, + reason, + target, + tuple(sorted({s.device for s in generated.slices})), + len(generated.slices), + NATIVE_FORMAT, + generated.omitted_devices, + ) + + def _import_compatible_dataset(self, options: ImportOptions) -> ImportResult: + capacity = options.events_per_slice or DEFAULT_CAPACITY + output = ( + (options.output_dir if options.output_dir is not None else options.snapshot_file.parent) + .expanduser() + .absolute() + ) + target = self._msinsight_backend.target_path(options.snapshot_file, output) + if target.resolve() != target or target.is_symlink(): + raise ImportExecutionError("Compatible output must be canonical and non-symlink.") + source_hash = self._hash_source(options.snapshot_file) + try: + cache_hash = salted_hash(options.snapshot_file) + except OSError as exc: + raise ImportExecutionError(f"Cannot calculate msinsight cacheHash: {exc}") from exc + if target.exists(): + try: + owned = inspect_owned_cache(target) + if ( + owned.identity != compatible_identity(source_hash, options.device, capacity) + or owned.manifest.cache_hash != cache_hash + or owned.manifest.source_file != str(options.snapshot_file.resolve()) + or owned.metadata.source_size != options.snapshot_file.stat().st_size + ): + raise ValueError("Source/options/format identity changed") + except ( + ValueError, + KeyError, + TypeError, + OSError, + sqlite3.DatabaseError, + DatabaseSchemaError, + ) as exc: + raise ImportExecutionError( + f"Existing compatible output is not reusable; preserved even under --force. Choose a new output directory: {exc}" + ) from exc + self._recheck_source(options.snapshot_file, source_hash) + return ImportResult( + target, + options.device, + self._set_focus_if_requested(target, options), + True, + owned.metadata, + None, + target, + tuple(sorted(d.device_id for d in owned.manifest.devices)), + sum(len(d.slices) for d in owned.manifest.devices), + COMPATIBILITY_FORMAT, + owned.omitted_devices, + ) + completed: ImportMetadata | None = None + generated: ShardedReplayResult | None = None + + def finalize(result: ShardedReplayResult) -> None: + nonlocal completed, generated + convert_shards(result) + metadata = replace( + self._metadata_service.build_metadata( + options.snapshot_file, source_hash, options.device + ), + import_format_version=1, + ) + for item in result.slices: + _require_readonly_member(result.directory, item.file) + self._metadata_service.write(item.file, metadata) + manifest = build_compatible_manifest( + result, metadata, options.snapshot_file, capacity, cache_hash + ) + with (result.directory / "manifest.json").open("x", encoding="utf-8") as file: + json.dump(manifest, file, indent=2) + validated = DatasetResolver().inspect(result.directory) + if validated is None or inspect_owned_cache(result.directory).metadata != metadata: + raise ImportExecutionError("Staged compatible artifact validation failed") + completed, generated = metadata, result + + focus_state: FocusState | None = None + + def focus(path: Path) -> None: + nonlocal focus_state + focus_state = self._set_focus_if_requested(path, options) + + self._msinsight_backend.dump_to_dataset( + options.snapshot_file, + target, + capacity=capacity, + device=options.device, + expected=None, + finalize=finalize, + post_publish=focus if options.set_focus else None, + pre_publish=lambda: self._recheck_source(options.snapshot_file, source_hash), + ) + assert completed is not None and generated is not None + return ImportResult( + target, + options.device, + focus_state, + False, + completed, + "database_missing", + target, + tuple(sorted({s.device for s in generated.slices})), + len(generated.slices), + COMPATIBILITY_FORMAT, + generated.omitted_devices, + ) + + def _hash_source(self, source: Path) -> str: + try: + return self._metadata_service.calculate_sha256(source) + except ImportMetadataError as exc: + raise ImportExecutionError(str(exc)) from exc + + def _recheck_source(self, source: Path, initial_hash: str) -> None: + if self._hash_source(source) != initial_hash: + raise SourceChangedError( + "Snapshot source changed while import was running; existing artifact preserved." + ) + def _set_focus_if_requested( self, db_path: Path, @@ -127,10 +404,17 @@ def _set_focus_if_requested( if self._focus_service is None: self._focus_service = FocusService() try: - return self._focus_service.set_project_focus( - db_path=db_path, - device_id=options.device, + transaction = ( + self._focus_service.project_focus_transaction() + if options.events_per_slice is not None + or options.format in (COMPATIBILITY_FORMAT, "msinsight") + else nullcontext() ) + with transaction: + return self._focus_service.set_project_focus( + db_path=db_path, + device_id=options.device, + ) except (FocusFileInvalidError, DatabaseSchemaError, InvalidDeviceError, OSError) as exc: raise ImportExecutionError( f"Imported database cannot be registered as focus: {exc}" diff --git a/src/pt_snap_cli/core/models.py b/src/pt_snap_cli/core/models.py index 0201e26..e994284 100644 --- a/src/pt_snap_cli/core/models.py +++ b/src/pt_snap_cli/core/models.py @@ -17,6 +17,7 @@ "import_format_changed", "device_changed", "forced", + "dataset_identity_changed", ] MetadataStatus = Literal["available", "unavailable", "invalid"] SplitFormat = Literal["pickle", "json"] @@ -93,6 +94,7 @@ class TemplateInfo: output_schema: list[dict[str, object]] | None semantics_version: int | None = None interpretation_limits: list[str] = field(default_factory=list) + dataset_support: dict[str, object] = field(default_factory=dict) @dataclass(frozen=True) @@ -107,6 +109,7 @@ class QueryResult: truncated: bool = False total_is_exact: bool = False timeout_s: float | None = None + scope: dict[str, object] | None = None @dataclass(frozen=True) @@ -125,6 +128,10 @@ class PeakMemoryReport: percent_denominator: str = "included_bytes" active_bytes_at_event: int | None = None coverage_percent: float | None = None + source_coverage: dict[str, object] | None = None + scope: dict[str, object] | None = None + timeout_s: float | None = None + budget_scope: str = "report_composition" @dataclass(frozen=True) @@ -133,8 +140,13 @@ class ImportOptions: Attributes: snapshot_file: Path to the input `.pkl` or `.pickle` snapshot. - output_dir: Directory where the generated `.db` is written. + output_dir: Parent of the generated standalone DB or native dataset. Defaults to the snapshot file's parent directory. + events_per_slice: Positive real-event capacity per device shard; None + preserves standalone behavior unless msinsight is explicit (500000). + Negative boundaries do not count. + format: None selects single-db or pt-snap-native-v2 from capacity; + msinsight/compatibility-v1 explicitly selects fixed-target inline v1. device: Optional device id to focus on. When None, all available devices are imported. set_focus: When True (default), also write project focus so subsequent @@ -146,6 +158,8 @@ class ImportOptions: device: int | None = None set_focus: bool = True force: bool = False + events_per_slice: int | None = None + format: str | None = None @dataclass(frozen=True) @@ -181,7 +195,9 @@ class ImportResult: """Result of `pt-snap import`. Attributes: - db_path: Path to the generated SQLite database. + db_path: Path to the standalone SQLite database or native dataset directory. + dataset_path: Native dataset directory, or None for standalone imports. + devices/slice_count/omitted_devices: Complete native output inventory. device_id: Device id used during import, or None when not specified. focus_state: The FocusState written to project focus when options.set_focus is True, otherwise None. @@ -193,6 +209,11 @@ class ImportResult: reused: bool metadata: ImportMetadata cache_miss_reason: CacheMissReason | None + dataset_path: Path | None = None + devices: tuple[int, ...] = () + slice_count: int = 0 + format: str = "single-db" + omitted_devices: tuple[int, ...] = () @dataclass(frozen=True) @@ -264,6 +285,7 @@ class DatabaseOverview: focus_source: FocusSource devices: list[DeviceTraceBounds] metadata: MetadataInspection + dataset: dict[str, object] | None = None @dataclass(frozen=True) diff --git a/src/pt_snap_cli/core/msinsight_export.py b/src/pt_snap_cli/core/msinsight_export.py new file mode 100644 index 0000000..834efeb --- /dev/null +++ b/src/pt_snap_cli/core/msinsight_export.py @@ -0,0 +1,258 @@ +"""Original interface-facts converter for fixed msinsight 101f65b8 compatibility. + +No Ascend implementation is copied (its source is Mulan PSL v2). Physical +schema/hash facts: blobs 1fe50a5136f85365db55f4632d58f3cf7090431e, +95612509ef990ac6b7034919dae4938cc6acf3ad and +ca421c7b33f3a0050df9e7a01e666f558801bc2f. See sharded-snapshotdb docs. +Only caller-owned finalized native staging is converted; existing caches are +never mutated, even under force. This is not GUI acceptance. +""" + +from __future__ import annotations + +import hashlib +import json +import sqlite3 +from contextlib import closing +from dataclasses import asdict, dataclass +from pathlib import Path + +from .dataset_contract import ( + ACTION_NAMES, + MANIFEST_SCHEMA_VERSION, + MSINSIGHT_REVISION, + TRACE_COLUMNS, + DatasetManifest, + _unique_object, + validate_dataset, + validate_event_ids, +) +from .dataset_files import hash_file, require_readonly_member +from .dataset_import_backend import DatasetImportBackend +from .errors import ImportExecutionError +from .import_metadata_contract import metadata_from_mapping +from .models import ImportMetadata +from .sharded_replay_service import ShardedReplayResult + +COMPATIBILITY_FORMAT = "compatibility-v1" +DEFAULT_CAPACITY = 500000 +SALT = b"mem_snapshot_parser_v2" +# Independent semantic versions, not the installed CLI version or GUI cacheHash. +EXPORT_VERSION = 1 +REFERENCE_VERSION = 1 + + +def salted_hash(source: Path) -> str: + digest = hashlib.sha256(SALT) + with source.open("rb") as file: + for chunk in iter(lambda: file.read(1024 * 1024), b""): + digest.update(chunk) + return digest.hexdigest() + + +def compatible_identity(source_hash: str, device: int | None, capacity: int) -> dict[str, object]: + return { + "sourceSha256": source_hash, + "requestedDevice": device, + "eventsPerSlice": capacity, + "outputFormat": COMPATIBILITY_FORMAT, + "manifestContractVersion": MANIFEST_SCHEMA_VERSION, + "exportContractVersion": EXPORT_VERSION, + "referenceContractVersion": REFERENCE_VERSION, + "metadataSchemaVersion": 1, + "importFormatVersion": 1, + "targetRevision": MSINSIGHT_REVISION, + } + + +class MsinsightImportBackend(DatasetImportBackend): + """Use the proven exclusive/compensating publisher, never its replacement path.""" + + @staticmethod + def target_path(source: Path, output_dir: Path) -> Path: + return output_dir / f"{source.name}.msinsight" + + @staticmethod + def inspect_target(target: Path) -> None: + if target.exists() or target.is_symlink(): + raise ImportExecutionError( + "Compatible output already exists; preserved even under --force. Choose a new output directory." + ) + return None + + +def convert_shards(result: ShardedReplayResult) -> None: + """Preserve event/stream/lifecycle values, materialize inline ninth column. + + First validate all selected IDs/actions, without inventing event zero or + coercing unsupported OOM. Block/reference tables are left byte-semantically + intact; no foreign real events are copied into the slice window. + """ + for device in sorted({s.device for s in result.slices}): + ids: list[int] = [] + for item in (s for s in result.slices if s.device == device): + require_readonly_member(result.directory, item.file) + with closing(sqlite3.connect(item.file.as_uri() + "?mode=ro", uri=True)) as conn: + ids.extend( + row[0] + for row in conn.execute( + f"SELECT id FROM trace_entry_{device} WHERE id>=0 ORDER BY id" + ) + ) + if ( + conn.execute( + f"SELECT id FROM trace_entry_{device} WHERE typeof(action)!='integer' " + "OR action NOT BETWEEN 0 AND 7 OR action IS NULL LIMIT 1" + ).fetchone() + is not None + ): + raise ValueError("msinsight compatibility does not support OOM/unknown actions") + validate_event_ids(ids) + for item in result.slices: + device = item.device + trace = f"trace_entry_{device}" + with closing(sqlite3.connect(item.file.as_uri() + "?mode=rw", uri=True)) as conn, conn: + # A missing stack reference must fail, not silently erase an event. + if ( + conn.execute( + f"SELECT t.id FROM {trace} t LEFT JOIN callstack c ON c.id=t.callstackId " + "WHERE c.id IS NULL LIMIT 1" + ).fetchone() + is not None + ): + raise ValueError("Missing finalized callstack reference") + conn.execute(f"ALTER TABLE {trace} RENAME TO pt_snap_native_trace") + columns = ",".join( + f'"{name}" {kind}' + (" PRIMARY KEY" if name == "id" else "") + for name, kind in TRACE_COLUMNS + ) + conn.execute(f"CREATE TABLE {trace} ({columns})") + conn.execute( + f"INSERT INTO {trace} SELECT t.id,t.action,t.address,t.size,t.stream," + "t.allocated,t.active,t.reserved,c.callstack FROM pt_snap_native_trace t " + "JOIN callstack c ON c.id=t.callstackId" + ) + conn.execute("DROP TABLE pt_snap_native_trace") + conn.execute("DROP TABLE callstack") + conn.execute('DELETE FROM dictionary WHERE "table"=? AND "column"=?', (trace, "action")) + conn.executemany( + "INSERT INTO dictionary VALUES (?,?,?,?)", + [(trace, "action", str(key), name) for key, name in enumerate(ACTION_NAMES)], + ) + for metric in ("allocated", "active", "reserved"): + conn.execute(f"CREATE INDEX {trace}_{metric} ON {trace} ({metric})") + + +def build_compatible_manifest( + result: ShardedReplayResult, + metadata: ImportMetadata, + source: Path, + capacity: int, + cache_hash: str, +) -> dict[str, object]: + devices: dict[str, object] = {} + files: dict[str, str] = {} + for device in sorted({s.device for s in result.slices}): + slices = [s for s in result.slices if s.device == device] + devices[str(device)] = { + "eventCount": sum(s.end_position - s.start_position + 1 for s in slices), + "sliceCount": len(slices), + "readySlices": list(range(len(slices))), + "slices": [ + { + "index": s.index, + "startEventId": s.start_event_id, + "endEventId": s.end_event_id, + "file": s.file.relative_to(result.directory).as_posix(), + "ready": True, + } + for s in slices + ], + } + for item in slices: + files[item.file.relative_to(result.directory).as_posix()] = hash_file(item.file) + return { + "schemaVersion": MANIFEST_SCHEMA_VERSION, + "status": "complete", + "sourceFile": str(source.resolve()), + "cacheHash": cache_hash, + "eventsPerSlice": capacity, + "devices": devices, + "ptSnap": { + "identity": compatible_identity( + metadata.source_sha256, metadata.requested_device, capacity + ), + "metadata": asdict(metadata), + "omittedDevices": list(result.omitted_devices), + "members": files, + }, + } + + +@dataclass(frozen=True) +class CompatibleCache: + manifest: DatasetManifest + identity: dict[str, object] + metadata: ImportMetadata + omitted_devices: tuple[int, ...] + + +def inspect_owned_cache(target: Path) -> CompatibleCache: + """Whole valid recognized identity only; salted hash alone is never ownership. + + Reject unexpected files/aliases rather than adopt them. GUI may change + derived tables: that invalidates pt-snap's immutable cache attestation, and + requires a new output, never repair or force-replacement. + """ + validation = validate_dataset(target) + with (target / "manifest.json").open(encoding="utf-8") as file: + data = json.load(file, object_pairs_hook=_unique_object) + extension = data["ptSnap"] + metadata = metadata_from_mapping(extension["metadata"]) + identity = extension["identity"] + expected_identity = compatible_identity( + metadata.source_sha256, metadata.requested_device, validation.manifest.events_per_slice + ) + if identity != expected_identity or any( + type(identity[key]) is not type(value) for key, value in expected_identity.items() + ): + raise ValueError("Unsupported or inconsistent pt-snap compatibility identity") + if ( + metadata.importer_name != "pt-snap-cli" + or metadata.import_format_version != 1 + or metadata.metadata_schema_version != 1 + ): + raise ValueError("Metadata must match physical v1 layout") + paths = {s.file for d in validation.manifest.devices for s in d.slices} + if set(extension["members"]) != paths: + raise ValueError("Incomplete member attestation") + expected = {"manifest.json", *paths, *(str(Path(p).parent) for p in paths)} + actual = {p.relative_to(target).as_posix() for p in target.rglob("*")} + if actual != expected or any(p.is_symlink() for p in target.rglob("*")): + raise ValueError("Unrecognized files or aliases in compatible cache") + for relative in paths: + member = target / relative + if hash_file(member) != extension["members"][relative]: + raise ValueError("Compatible member hash changed") + with closing(sqlite3.connect(member.as_uri() + "?mode=ro&immutable=1", uri=True)) as conn: + conn.row_factory = sqlite3.Row + rows = conn.execute("SELECT * FROM pt_snap_metadata").fetchall() + if ( + len(rows) != 1 + or rows[0]["id"] != 1 + or metadata_from_mapping(dict(rows[0])) != metadata + ): + raise ValueError("Member metadata disagrees with identity") + omitted = extension["omittedDevices"] + devices = {d.device_id for d in validation.manifest.devices} + if ( + not isinstance(omitted, list) + or any(type(v) is not int or v < 0 or v in devices for v in omitted) + or len(set(omitted)) != len(omitted) + ): + raise ValueError("Invalid omitted device inventory") + if metadata.requested_device is not None and devices != {metadata.requested_device}: + raise ValueError("Device selection disagrees with identity") + if Path(validation.manifest.source_file).name != metadata.source_name: + raise ValueError("Source path disagrees with metadata") + return CompatibleCache(validation.manifest, identity, metadata, tuple(omitted)) diff --git a/src/pt_snap_cli/core/native_dataset_contract.py b/src/pt_snap_cli/core/native_dataset_contract.py new file mode 100644 index 0000000..729269f --- /dev/null +++ b/src/pt_snap_cli/core/native_dataset_contract.py @@ -0,0 +1,577 @@ +"""Original published pt-snap native-v2 protocol; NOT compatibility-v1/msinsight. + +Counts/capacity use original positions per device. IDs stay sparse and original. +Only closed, finalized artifacts are accepted; unknown extensions grant no capabilities. +""" + +from __future__ import annotations + +import json +import re +import sqlite3 +from contextlib import closing +from dataclasses import asdict, dataclass +from pathlib import Path +from typing import cast + +from .dataset_contract import ( + ACTION_NAMES, + BLOCK_COLUMNS, + BLOCK_STATES, + DICTIONARY_COLUMNS, + TRACE_COLUMNS, + DatasetContractError, + DatasetManifest, + DatasetValidation, + DeviceRecord, + QueryScope, + SliceRecord, + _array, + _integer, + _need, + _object, + _safe_file, + _schema, + _text, + _unique_object, +) +from .dataset_files import hash_file as _hash_file +from .dataset_files import require_readonly_member as _require_readonly_member +from .import_metadata_contract import metadata_from_mapping +from .models import ImportMetadata +from .sharded_replay_service import ShardedReplayResult + +NATIVE_FORMAT = "pt-snap-native-v2" +NATIVE_MANIFEST_VERSION = 1 +# Semantic contracts, deliberately independent of the installed CLI __version__. +NATIVE_FORMAT_VERSION = 1 +NATIVE_IMPORT_VERSION = 1 +NATIVE_EXTENSION_VERSION = 1 + + +@dataclass(frozen=True) +class NativeSlice(SliceRecord): + start_position: int + end_position: int + sha256: str + + +@dataclass(frozen=True) +class NativeManifest(DatasetManifest): + identity: dict[str, object] + metadata: ImportMetadata + omitted_devices: tuple[int, ...] + + +def cache_identity(source_sha256: str, device: int | None, capacity: int) -> dict[str, object]: + return { + "sourceSha256": source_sha256, + "requestedDevice": device, + "eventsPerSlice": capacity, + "outputFormat": NATIVE_FORMAT, + "manifestContractVersion": NATIVE_MANIFEST_VERSION, + "formatContractVersion": NATIVE_FORMAT_VERSION, + "importContractVersion": NATIVE_IMPORT_VERSION, + "extensionContractVersion": NATIVE_EXTENSION_VERSION, + "metadataSchemaVersion": 1, + "importFormatVersion": 2, + } + + +def _digest(value: object, loc: str) -> str: + result = _text(value, loc) + _need(re.fullmatch(r"[0-9a-f]{64}", result) is not None, loc, "hash", "expected SHA256") + return result + + +def parse_native_manifest(value: object) -> NativeManifest: + + data = _object(value, "manifest") + _need(data.get("format") == NATIVE_FORMAT, "format", "format", "unsupported native format") + version = _integer(data.get("schemaVersion"), "schemaVersion", minimum=1) + _need( + version == NATIVE_MANIFEST_VERSION, + "schemaVersion", + "version", + "unsupported native manifest", + ) + _need( + data.get("status") == "complete", "status", "building", "complete native dataset required" + ) + source = _text(data.get("sourceFile"), "sourceFile") + identity = _object(data.get("identity"), "identity") + _digest(identity.get("sourceSha256"), "identity.sourceSha256") + requested = identity.get("requestedDevice") + if requested is not None: + _integer(requested, "identity.requestedDevice", maximum=2**31 - 1) + capacity = _integer(identity.get("eventsPerSlice"), "identity.eventsPerSlice", minimum=1) + _need( + identity.get("outputFormat") == NATIVE_FORMAT, + "identity", + "format", + "format identity mismatch", + ) + for key in ( + "manifestContractVersion", + "formatContractVersion", + "importContractVersion", + "extensionContractVersion", + "metadataSchemaVersion", + "importFormatVersion", + ): + _integer(identity.get(key), "identity." + key, minimum=1) + _need( + set(identity) == set(cache_identity("", None, 1)), + "identity", + "fields", + "identity fields differ", + ) + try: + metadata = metadata_from_mapping(_object(data.get("metadata"), "metadata")) + except (KeyError, TypeError, ValueError) as exc: + raise DatasetContractError("metadata", "metadata", str(exc)) from exc + _need( + ( + metadata.source_sha256, + metadata.requested_device, + metadata.metadata_schema_version, + metadata.import_format_version, + ) + == ( + identity["sourceSha256"], + requested, + identity["metadataSchemaVersion"], + identity["importFormatVersion"], + ), + "metadata", + "identity", + "metadata disagrees with dataset identity", + ) + _need( + source == metadata.source_name, + "sourceFile", + "metadata", + "source name disagrees with metadata", + ) + omitted = tuple( + _integer(v, "omittedDevices", maximum=2**31 - 1) + for v in _array(data.get("omittedDevices"), "omittedDevices") + ) + _need(len(set(omitted)) == len(omitted), "omittedDevices", "duplicate", "duplicate device") + devices: list[DeviceRecord] = [] + for key, raw in _object(data.get("devices"), "devices").items(): + loc = f"devices.{key}" + _need( + isinstance(key, str) and re.fullmatch(r"0|[1-9][0-9]*", key) is not None, + loc, + "device", + "noncanonical device", + ) + device_id = _integer(int(key), loc, maximum=2**31 - 1) + device = _object(raw, loc) + count = _integer(device.get("eventCount"), loc + ".eventCount", minimum=1) + records = _array(device.get("slices"), loc + ".slices") + _need( + bool(records) + and device.get("sliceCount") == len(records) + and type(device.get("sliceCount")) is int, + loc, + "count", + "invalid sliceCount", + ) + ready = tuple( + _integer(v, loc + ".readySlices") + for v in _array(device.get("readySlices"), loc + ".readySlices") + ) + _need( + ready == tuple(range(len(records))), + loc, + "readiness", + "all slices must be ready in order", + ) + slices: list[SliceRecord] = [] + position, last_id = 0, -1 + for index, raw_slice in enumerate(records): + item = _object(raw_slice, loc) + _need( + type(item.get("index")) is int + and item["index"] == index + and item.get("ready") is True, + loc, + "index", + "slice index/readiness mismatch", + ) + start = _integer(item.get("startPosition"), loc + ".startPosition") + end = _integer(item.get("endPosition"), loc + ".endPosition", minimum=start) + _need( + start == position and end - start + 1 <= capacity, + loc, + "coverage", + "invalid positional coverage/capacity", + ) + # All but the last shard must be full; no omitted positions are allowed. + _need( + index == len(records) - 1 or end - start + 1 == capacity, + loc, + "capacity", + "nonfinal slice is not full", + ) + low = _integer(item.get("startEventId"), loc + ".startEventId", minimum=last_id + 1) + high = _integer(item.get("endEventId"), loc + ".endEventId", minimum=low) + file = _text(item.get("file"), loc + ".file") + _need( + file == f"device_{device_id}/slice_{index:05d}.db", + loc, + "path", + "noncanonical member", + ) + slices.append( + NativeSlice( + index, low, high, file, True, start, end, _digest(item.get("sha256"), loc) + ) + ) + position, last_id = end + 1, high + _need(position == count, loc, "coverage", "positions do not cover eventCount") + devices.append(DeviceRecord(device_id, count, ready, tuple(slices))) + _need(bool(devices), "devices", "empty", "event-bearing device required") + selected = {d.device_id for d in devices} + _need( + not selected.intersection(omitted), "omittedDevices", "device", "selected device is omitted" + ) + _need( + requested is None or selected == {requested}, + "identity", + "device", + "requested device mismatch", + ) + return NativeManifest( + version, "complete", source, "", capacity, tuple(devices), identity, metadata, omitted + ) + + +def read_native_manifest(root: Path) -> NativeManifest: + try: + with _safe_file(root, "manifest.json").open(encoding="utf-8") as file: + return parse_native_manifest(json.load(file, object_pairs_hook=_unique_object)) + except (OSError, UnicodeError, json.JSONDecodeError) as exc: + raise DatasetContractError("manifest.json", "read", str(exc)) from exc + + +def require_native_members(root: Path, manifest: NativeManifest) -> None: + """Ownership recognition before any replacement: never adopt unknown files/dirs.""" + _need( + root.resolve() == root and not root.is_symlink(), + "dataset", + "path", + "noncanonical directory", + ) + expected = {"manifest.json", *(f"device_{d.device_id}" for d in manifest.devices)} + _need( + {p.name for p in root.iterdir()} == expected, + "dataset", + "members", + "unexpected/missing dataset member", + ) + for device in manifest.devices: + directory = root / f"device_{device.device_id}" + _need( + not directory.is_symlink() and directory.is_dir(), + str(directory), + "path", + "unsafe device directory", + ) + _need( + {p.name for p in directory.iterdir()} == {Path(s.file).name for s in device.slices}, + str(directory), + "members", + "unexpected/missing shard", + ) + for item in device.slices: + _safe_file(root, item.file) + + +def validate_native_dataset(root: Path, manifest: NativeManifest) -> DatasetValidation: + + require_native_members(root, manifest) + expected_identity = cache_identity( + manifest.metadata.source_sha256, + manifest.metadata.requested_device, + manifest.events_per_slice, + ) + _need( + manifest.identity == expected_identity, + "identity", + "version", + "unsupported native semantic contract", + ) + # Check EVERY member before the FIRST SQLite open. No WAL/sidecars/repair. + for device in manifest.devices: + for raw in device.slices: + item = cast(NativeSlice, raw) + path = root / item.file + _require_readonly_member(root, path) + _need(_hash_file(path) == item.sha256, item.file, "hash", "member content changed") + boundaries = 0 + for device in manifest.devices: + ids: set[int] = set() + lifetimes: dict[int, tuple[object, ...]] = {} + references: list[tuple[int, int]] = [] + for raw in device.slices: + item = cast(NativeSlice, raw) + path = root / item.file + try: + with closing(sqlite3.connect(path.as_uri() + "?mode=ro", uri=True)) as conn: + conn.execute("PRAGMA query_only=ON") + _need( + conn.execute("PRAGMA quick_check").fetchall() == [("ok",)], + item.file, + "sqlite", + "integrity check failed", + ) + trace, block = f"trace_entry_{device.device_id}", f"block_{device.device_id}" + names = { + row[0] + for row in conn.execute( + "SELECT name FROM sqlite_master WHERE type IN ('table','view')" + ) + } + _need( + {n for n in names if n.lower().startswith(("trace_entry_", "block_"))} + == {trace, block}, + item.file, + "device_tables", + "wrong device/table set", + ) + for table, columns in ( + (trace, TRACE_COLUMNS[:-1] + (("callstackId", "INTEGER"),)), + (block, BLOCK_COLUMNS), + ("dictionary", DICTIONARY_COLUMNS), + ("callstack", (("id", "INTEGER"), ("callstack", "TEXT"))), + ): + _schema(conn, table, columns, item.file) + _need( + conn.execute( + "SELECT type FROM sqlite_master WHERE name='pt_snap_block_reference'" + ).fetchone() + == ("table",), + item.file, + "extension", + "finalized reference table required", + ) + reference_columns = conn.execute( + "PRAGMA table_info(pt_snap_block_reference)" + ).fetchall() + _need( + tuple((r[1], r[2].upper(), r[5]) for r in reference_columns) + == ( + ("blockId", "INTEGER", 1), + ("stream", "INTEGER", 0), + ("allocCallstack", "TEXT", 0), + ("freeCallstack", "TEXT", 0), + ), + item.file, + "extension", + "invalid reference schema", + ) + for table, column, values in ( + (trace, "action", tuple(enumerate((*ACTION_NAMES, "oom")))), + (block, "state", BLOCK_STATES), + ): + rows = conn.execute( + 'SELECT "key","value" FROM dictionary WHERE "table"=? AND "column"=?', + (table, column), + ).fetchall() + expected = {(str(k), v) for k, v in values} + _need( + len(rows) == len(expected) and set(rows) == expected, + item.file, + "dictionary", + "invalid native dictionary", + ) + real = [ + r[0] + for r in conn.execute(f'SELECT id FROM "{trace}" WHERE id>=0 ORDER BY id') + ] + _need( + len(real) == item.end_position - item.start_position + 1 + and (real[0], real[-1]) == (item.start_event_id, item.end_event_id) + and not ids.intersection(real), + item.file, + "event_identity", + "real IDs/count differ from positions/ranges", + ) + ids.update(real) + numeric = " OR ".join( + f"typeof({name})!='integer'" for name, _ in TRACE_COLUMNS[:-1] + ) + invalid = conn.execute( + f"SELECT id FROM \"{trace}\" WHERE {numeric} OR action NOT BETWEEN 0 AND 8 OR (id<0 AND action NOT IN (0,2)) OR size<0 OR allocated<0 OR active<0 OR reserved<0 OR typeof(callstackId)!='integer' OR callstackId NOT IN (SELECT id FROM callstack) LIMIT 1" + ).fetchone() + _need( + invalid is None, + item.file, + "event_values", + "invalid native event/scalar/callstack", + ) + _need( + conn.execute( + "SELECT id FROM callstack WHERE typeof(callstack) NOT IN ('text','null') LIMIT 1" + ).fetchone() + is None, + item.file, + "callstack", + "invalid text stack", + ) + boundaries += conn.execute( + f'SELECT COUNT(*) FROM "{trace}" WHERE id<0' + ).fetchone()[0] + stacks = { + r[0]: r[1:] for r in conn.execute("SELECT * FROM pt_snap_block_reference") + } + blocks = conn.execute(f'SELECT * FROM "{block}"').fetchall() + _need( + set(stacks) == {r[0] for r in blocks}, + item.file, + "extension", + "reference/block membership differs", + ) + for row in blocks: + bid, address, size, requested, state, alloc, free = row + _need( + all(type(v) is int for v in row) + and state in (-1, 0, 1) + and size >= 0 + and 0 <= requested <= size + and ((alloc == -1 and bid < 0) or (alloc >= 0 and bid == alloc)) + and (free == -1 or free >= 0 and (alloc == -1 or free >= alloc)), + item.file, + "block_identity", + "unfinalized/invalid block", + ) + ref = stacks[bid] + _need( + type(ref[0]) is int + and all(v is None or isinstance(v, str) for v in ref[1:]), + item.file, + "extension", + "invalid reference values", + ) + signature = (address, size, requested, alloc, free, *ref) + _need( + bid not in lifetimes or lifetimes[bid] == signature, + item.file, + "block_identity", + "conflicting finalized identity/reference", + ) + lifetimes[bid] = signature + references.append((alloc, free)) + except sqlite3.DatabaseError as exc: + raise DatasetContractError(item.file, "sqlite", str(exc)) from exc + try: + with closing( + sqlite3.connect(path.as_uri() + "?mode=ro", uri=True) + ) as metadata_conn: + metadata_conn.row_factory = sqlite3.Row + _need( + metadata_conn.execute( + "SELECT type FROM sqlite_master WHERE name='pt_snap_metadata'" + ).fetchone()[0] + == "table", + item.file, + "metadata", + "metadata must be an actual table", + ) + rows = metadata_conn.execute("SELECT * FROM pt_snap_metadata").fetchall() + _need( + len(rows) == 1 + and rows[0]["id"] == 1 + and set(rows[0].keys()) == {"id", *asdict(manifest.metadata)}, + item.file, + "metadata", + "invalid metadata row/schema", + ) + metadata = metadata_from_mapping(dict(rows[0])) + _need( + metadata == manifest.metadata, + item.file, + "metadata", + "member metadata differs from manifest", + ) + except (sqlite3.DatabaseError, KeyError, TypeError, ValueError) as exc: + raise DatasetContractError(item.file, "metadata", str(exc)) from exc + _require_readonly_member(root, path) + _need( + _hash_file(path) == item.sha256, + item.file, + "hash", + "member changed during validation", + ) + _need( + len(ids) == device.event_count + and all(v == -1 or v in ids for pair in references for v in pair), + "devices", + "event_identity", + "count/lifetime references not in original device scope", + ) + return DatasetValidation(manifest, sum(d.event_count for d in manifest.devices), boundaries) + + +def validate_native_scope(scope: QueryScope, manifest: NativeManifest) -> None: + if scope.kind == "event_range": + device = next((d for d in manifest.devices if d.device_id == scope.device_id), None) + _need( + type(scope.device_id) is int and device is not None, "scope", "device", "unknown device" + ) + assert device is not None + _need(scope.slice_index is None, "scope", "selector", "unexpected slice selector") + start = _integer( + scope.start_event_id, + "scope.start_event_id", + minimum=device.slices[0].start_event_id, + maximum=device.slices[-1].end_event_id, + ) + _integer( + scope.end_event_id, + "scope.end_event_id", + minimum=start, + maximum=device.slices[-1].end_event_id, + ) + else: + scope.validate(manifest) + + +def build_native_manifest( + result: ShardedReplayResult, metadata: ImportMetadata, capacity: int +) -> dict[str, object]: + + devices: dict[str, object] = {} + for device in sorted({item.device for item in result.slices}): + slices = [s for s in result.slices if s.device == device] + devices[str(device)] = { + "eventCount": sum(s.end_position - s.start_position + 1 for s in slices), + "sliceCount": len(slices), + "readySlices": list(range(len(slices))), + "slices": [ + { + "index": s.index, + "startPosition": s.start_position, + "endPosition": s.end_position, + "startEventId": s.start_event_id, + "endEventId": s.end_event_id, + "file": s.file.relative_to(result.directory).as_posix(), + "ready": True, + "sha256": _hash_file(s.file), + } + for s in slices + ], + } + return { + "format": NATIVE_FORMAT, + "schemaVersion": NATIVE_MANIFEST_VERSION, + "status": "complete", + "sourceFile": metadata.source_name, + "identity": cache_identity(metadata.source_sha256, metadata.requested_device, capacity), + "metadata": asdict(metadata), + "omittedDevices": list(result.omitted_devices), + "devices": devices, + } diff --git a/src/pt_snap_cli/core/overview_service.py b/src/pt_snap_cli/core/overview_service.py index ea974ef..c8be173 100644 --- a/src/pt_snap_cli/core/overview_service.py +++ b/src/pt_snap_cli/core/overview_service.py @@ -6,6 +6,7 @@ from pt_snap_cli.context import Context, DatabaseNotFoundError, SchemaVersionError from pt_snap_cli.core.context_cache import ContextCache +from pt_snap_cli.core.dataset_resolver import DatasetResolver from pt_snap_cli.core.errors import ( DatabaseMissingError, DatabaseSchemaError, @@ -48,7 +49,25 @@ def inspect( if resolved.db_path is None: raise FocusNotConfiguredError("No database path specified and no database configured.") + dataset = DatasetResolver().inspect(resolved.db_path) db_path = resolved.db_path.expanduser().resolve() + if dataset is not None: + return DatabaseOverview( + db_path=db_path, + focus_source=resolved.source, + devices=[ + DeviceTraceBounds( + device.device_id, + device.slices[0].start_event_id, + device.slices[-1].end_event_id, + ) + for device in sorted( + dataset.validation.manifest.devices, key=lambda d: d.device_id + ) + ], + metadata=self._metadata_service.inspect(db_path), + dataset=dataset.to_dict(), + ) ctx = self._validated_context(db_path) try: raw_bounds = ctx.device_trace_bounds() @@ -74,6 +93,7 @@ def overview_to_dict(self, overview: DatabaseOverview) -> dict[str, Any]: return { "db_path": str(overview.db_path), "focus_source": overview.focus_source, + **({"dataset": overview.dataset} if overview.dataset is not None else {}), "devices": [ { "device_id": device.device_id, diff --git a/src/pt_snap_cli/core/query_service.py b/src/pt_snap_cli/core/query_service.py index d0ff782..138e342 100644 --- a/src/pt_snap_cli/core/query_service.py +++ b/src/pt_snap_cli/core/query_service.py @@ -9,6 +9,12 @@ from pt_snap_cli.context import Context, DatabaseNotFoundError, SchemaVersionError from pt_snap_cli.core.context_cache import ContextCache +from pt_snap_cli.core.dataset_attribution import event_attribution, summarize_dataset_stacks +from pt_snap_cli.core.dataset_contract import QueryScope +from pt_snap_cli.core.dataset_global import global_query +from pt_snap_cli.core.dataset_resolver import DatasetResolver +from pt_snap_cli.core.dataset_sources import DatasetSourceResolver, QueryBudget +from pt_snap_cli.core.dataset_support import dataset_support from pt_snap_cli.core.errors import ( DatabaseMissingError, DatabaseSchemaError, @@ -32,6 +38,7 @@ discover_categories, get_query, get_template_info, + is_packaged_query, list_by_category_with_details, list_queries_with_details, ) @@ -156,6 +163,11 @@ def get_template_info(self, name: str) -> TemplateInfo: for param_name, param_details in info["parameters"].items() }, output_schema=copy.deepcopy(info["output_schema"]), + dataset_support=dataset_support( + name, + is_packaged_query(name) + and not (self._executor is not None and self._executor.has_runtime_template(name)), + ), semantics_version=info.get("semantics_version"), interpretation_limits=( [str(item) for item in info["interpretation_limits"]] @@ -189,6 +201,7 @@ def template_info_to_dict(info: TemplateInfo) -> dict[str, Any]: "output_schema": info.output_schema, "semantics_version": info.semantics_version, "interpretation_limits": list(info.interpretation_limits), + "dataset_support": dict(info.dataset_support), } def execute_query( @@ -202,7 +215,16 @@ def execute_query( *, exact_total: bool = False, timeout_s: float | None = None, + slice_index: int | None = None, + _budget: QueryBudget | None = None, ) -> QueryResult: + budget = ( + _budget + if _budget is not None + else QueryBudget(resolve_query_timeout(timeout_s), time.monotonic()) + ) + timeout_s, started = budget.timeout_s, budget.started + budget.remaining() resolved = self._focus_service.resolve_focus( explicit_db_path=db_path, explicit_device_id=device_id, @@ -211,12 +233,62 @@ def execute_query( if resolved.db_path is None: raise FocusNotConfiguredError("No database path specified and no database configured.") - ctx = self._validated_context(resolved.db_path) - target_device = self._resolve_device_id(ctx, resolved.device_id, device_id) - executor = self._get_executor(ctx) + dataset = DatasetResolver().inspect(resolved.db_path) + budget.remaining() + scope: dict[str, object] | None = None query_params = params or {} - timeout_s = resolve_query_timeout(timeout_s) - started = time.monotonic() + sources: DatasetSourceResolver | None = None + if dataset is not None: + selected = dataset.device(device_id if device_id is not None else resolved.device_id) + sources = DatasetSourceResolver( + dataset, selected.device_id, self._context_cache, budget + ) + template_obj = get_query(template) + if template_obj is None: + raise TemplateNotFoundError(f"Template '{template}' not found") + support = dataset_support(template, is_packaged_query(template)) + overridden = self._executor is not None and self._executor.has_runtime_template( + template + ) + if not support["supported"] or overridden: + raise QueryExecutionError( + "Unsupported dataset-global template/custom SQL or runtime override; " + "select an explicit standalone member. A built-in name is not a merge contract." + ) + for name in ("id", "min_id", "max_id", "start_id", "end_id", "event_id"): + if isinstance(query_params.get(name), bool): + raise InvalidParameterError("Event/block endpoints must not be booleans.") + try: + validated = template_obj.validate_params(query_params) + except (TypeError, ValueError) as exc: + raise TemplateRenderError(str(exc)) from exc + if slice_index is not None: + dataset.paths(QueryScope("slice", selected.device_id, slice_index)) + implementation = ( + event_attribution + if template in ("active_blocks_at_event", "active_memory_callstack_at_event") + else global_query + ) + result = implementation( + sources, + template, + validated, + max_rows, + exact_total, + slice_index, + template_obj.semantics_version, + ) + stack_bytes = validated.get("stack_bytes") + if isinstance(stack_bytes, int) and stack_bytes >= 0: + summarize_dataset_stacks(result.rows, stack_bytes) + budget.remaining() + return result + else: + if slice_index is not None: + raise InvalidParameterError("slice_index requires a manifest dataset.") + ctx = self._validated_context(resolved.db_path) + target_device = self._resolve_device_id(ctx, resolved.device_id, device_id) + executor = self._get_executor(ctx) try: rows, has_more, _applied_limit = executor.execute_template_page( @@ -224,7 +296,7 @@ def execute_query( query_params, device_id=target_device, max_rows=max_rows, - timeout_s=timeout_s, + timeout_s=budget.remaining(), ) offset = _validated_offset(template, query_params) rank_cap = _finite_top_n(template, query_params) @@ -249,7 +321,7 @@ def execute_query( except ExecutorTemplateRenderError as exc: raise TemplateRenderError(str(exc)) from exc except ExecutorQueryTimeoutError as exc: - raise QueryTimeoutError(str(exc)) from exc + raise QueryTimeoutError(f"Query timed out after {timeout_s} seconds") from exc except ExecutorQueryExecutionError as exc: if get_query(template) is None: raise TemplateNotFoundError(f"Template '{template}' not found") from exc @@ -259,7 +331,11 @@ def execute_query( if template_obj is not None and "stack_bytes" in template_obj.parameters: stack_bytes = template_obj.validate_params(query_params)["stack_bytes"] if isinstance(stack_bytes, int) and stack_bytes >= 0: - summarize_stacks(rows, stack_bytes, ctx.callstack_layout) + if sources is not None: + summarize_dataset_stacks(rows, stack_bytes) + else: + summarize_stacks(rows, stack_bytes, ctx.callstack_layout) + budget.remaining() return QueryResult( total=total, returned=returned, @@ -271,6 +347,7 @@ def execute_query( truncated=truncated, total_is_exact=total_is_exact, timeout_s=timeout_s, + scope=scope, ) def _get_executor(self, ctx: Context) -> QueryExecutor: @@ -308,8 +385,12 @@ def _resolve_device_id( raise InvalidDeviceError("No devices found in database.") return ctx.device_ids[0] - def _validated_context(self, db_path: Path) -> Context: + def _validated_context( + self, db_path: Path, *, generation: str | None = None, immutable: bool = False + ) -> Context: try: + if generation is not None: + return self._context_cache.get(db_path, generation=generation, immutable=immutable) return self._context_cache.get(db_path) except DatabaseNotFoundError as exc: raise DatabaseMissingError(str(exc)) from exc diff --git a/src/pt_snap_cli/core/report_service.py b/src/pt_snap_cli/core/report_service.py index 7761eb7..69a3090 100644 --- a/src/pt_snap_cli/core/report_service.py +++ b/src/pt_snap_cli/core/report_service.py @@ -1,11 +1,13 @@ from __future__ import annotations +import time from pathlib import Path from typing import Literal, cast +from pt_snap_cli.core.dataset_sources import QueryBudget from pt_snap_cli.core.focus_service import FocusService -from pt_snap_cli.core.models import PeakMemoryReport -from pt_snap_cli.core.query_service import QueryService +from pt_snap_cli.core.models import PeakMemoryReport, QueryResult +from pt_snap_cli.core.query_service import QueryService, resolve_query_timeout PeakMetric = Literal["active", "allocated", "reserved"] @@ -31,6 +33,40 @@ def close(self) -> None: """Release connections owned by this report service; allow later reuse.""" self._query_service.close() + def event_attribution( + self, + event_id: int, + db_path: Path | str | None = None, + device_id: int | None = None, + *, + include_static: bool = True, + limit: int = 20, + stack_bytes: int = -1, + start_dir: Path | None = None, + timeout_s: float | None = None, + _budget: QueryBudget | None = None, + ) -> QueryResult: + """Reuse core point-event attribution, also for externally selected peaks. + + Dataset-global peak selection is a separate contract; this method does + not silently choose a local shard peak or invent an event for empty data. + """ + return self._query_service.execute_query( + "active_memory_callstack_at_event", + params={ + "event_id": event_id, + "include_static": include_static, + "min_size": 0, + "top_n": limit, + "stack_bytes": stack_bytes, + }, + db_path=db_path, + device_id=device_id, + start_dir=start_dir, + timeout_s=timeout_s, + _budget=_budget, + ) + def peak_memory_report( self, db_path: Path | str | None = None, @@ -40,14 +76,22 @@ def peak_memory_report( limit: int = 20, start_dir: Path | None = None, stack_bytes: int = -1, + *, + start_id: int | None = None, + end_id: int | None = None, + timeout_s: float | None = None, ) -> PeakMemoryReport: if metric not in _EVENT_ID_BY_METRIC: raise ValueError( f"Invalid metric '{metric}'. Must be one of: active, allocated, reserved" ) + budget = QueryBudget(resolve_query_timeout(timeout_s), time.monotonic()) + range_params = {"start_id": start_id, "end_id": end_id} peak_result = self._query_service.execute_query( "memory_peak", + params=range_params, + _budget=budget, db_path=db_path, device_id=device_id, start_dir=start_dir, @@ -73,20 +117,27 @@ def peak_memory_report( callstack_groups=[], total_is_exact=True, effective_params=attribution_params, + scope=peak_result.scope, + timeout_s=budget.timeout_s, ) gap_result = self._query_service.execute_query( "allocator_gap", + params=range_params, + _budget=budget, db_path=db_path, device_id=device_id, start_dir=start_dir, ) - callstack_result = self._query_service.execute_query( - "active_memory_callstack_at_event", - params=attribution_params, - db_path=db_path, - device_id=device_id, + callstack_result = self.event_attribution( + event_id, + db_path, + device_id, + include_static=include_static, + limit=limit, + stack_bytes=stack_bytes, start_dir=start_dir, + _budget=budget, ) gap = gap_result.rows[0] if gap_result.rows else None @@ -116,4 +167,9 @@ def peak_memory_report( included_bytes=included_bytes, active_bytes_at_event=active_bytes, coverage_percent=coverage, + scope=peak_result.scope, + timeout_s=budget.timeout_s, + source_coverage=cast( + dict[str, object] | None, (callstack_result.scope or {}).get("source_coverage") + ), ) diff --git a/src/pt_snap_cli/core/sharded_replay_service.py b/src/pt_snap_cli/core/sharded_replay_service.py new file mode 100644 index 0000000..bb210c6 --- /dev/null +++ b/src/pt_snap_cli/core/sharded_replay_service.py @@ -0,0 +1,57 @@ +"""Internal P1 shared producer boundary, not a CLI import/export facade.""" + +from __future__ import annotations + +from dataclasses import dataclass +from pathlib import Path +from typing import cast + +from ..snapshot.representation import load_snapshot_representation +from ..snapshot.tools.adaptors.sharded_replay import ReplaySlice, build_shards + + +@dataclass(frozen=True) +class ShardedReplayResult: + directory: Path + slices: tuple[ReplaySlice, ...] + omitted_devices: tuple[int, ...] + layout: str = "native-v2" + + +class ShardedReplayService: + def stage( + self, + source: Path | str, + directory: Path | str, + *, + events_per_slice: int, + device: int | None = None, + ) -> ShardedReplayResult: + """Load once and finalize native shards in a NEW private staging directory. + + The caller owns partial files on failure. No manifest, ready slice, focus, + cache reuse or final publication is exposed. Only a successful result can + be handed to a subsequent exporter/publication service. Pickle is trusted + executable input; this does not implement streaming or an RSS bound. + """ + if type(events_per_slice) is not int or events_per_slice <= 0: + raise ValueError("events_per_slice must be a positive integer") + if device is not None and (type(device) is not int or device < 0): + raise ValueError("device must be a nonnegative integer") + target = Path(directory).absolute() + if target.resolve() != target or target.is_symlink(): + raise ValueError("Staging directory must be canonical and non-symlink") + if target.exists(): + raise FileExistsError(target) + data = load_snapshot_representation(Path(source)) + traces = cast(list[list[object]], data.get("device_traces", [])) + available = tuple(index for index, entries in enumerate(traces) if entries) + if not available or (device is not None and device not in available): + raise ValueError("Requested device is absent or has no trace events") + selected = available if device is None else (device,) + # Exclusive allocation: never replace or adopt an existing directory. + target.mkdir() + slices = build_shards(data, target, events_per_slice, selected) + return ShardedReplayResult( + target, slices, tuple(index for index in range(len(traces)) if index not in selected) + ) diff --git a/src/pt_snap_cli/query/executor.py b/src/pt_snap_cli/query/executor.py index 7537bc0..1704c3d 100644 --- a/src/pt_snap_cli/query/executor.py +++ b/src/pt_snap_cli/query/executor.py @@ -5,7 +5,7 @@ import re import sqlite3 import time -from collections.abc import Generator, Mapping +from collections.abc import Callable, Generator, Mapping from contextlib import contextmanager from pathlib import Path from typing import Any @@ -326,6 +326,8 @@ def execute( sql: str, params: list[Any] | None = None, timeout_s: float | None = None, + *, + consume: Callable[[list[dict[str, Any]]], None] | None = None, ) -> list[dict[str, Any]]: """Execute raw SQL query. @@ -345,12 +347,26 @@ def execute( with self._context.connect() as conn: with _sqlite_timeout(conn, timeout_s): cursor = conn.cursor() - if params: - cursor.execute(sql, params) - else: - cursor.execute(sql) - columns = [desc[0] for desc in cursor.description] - return [dict(zip(columns, row, strict=True)) for row in cursor.fetchall()] + try: + if params: + cursor.execute(sql, params) + else: + cursor.execute(sql) + columns = [desc[0] for desc in cursor.description] + if consume is None: + # Preserve standalone behavior and its established + # cursor contract. Dataset reads explicitly opt in. + return [ + dict(zip(columns, row, strict=True)) for row in cursor.fetchall() + ] + result = [] + while batch := cursor.fetchmany(256): + rows = [dict(zip(columns, row, strict=True)) for row in batch] + consume(rows) + result.extend(rows) + return result + finally: + cursor.close() except sqlite3.OperationalError as e: if timeout_s is not None and timeout_s > 0 and _is_sqlite_interrupt(e): raise self._timeout_error(timeout_s) from e @@ -570,6 +586,10 @@ def register_template(self, template: QueryTemplate, config_name: str = "default self._configs[config_name] = QueryConfig() self._configs[config_name].queries[template.name] = template + def has_runtime_template(self, name: str) -> bool: + """A local override cannot inherit a packaged dataset merge contract.""" + return any(name in config.queries for config in self._configs.values()) + def list_templates(self) -> list[str]: """List all available template names. diff --git a/src/pt_snap_cli/query/registry.py b/src/pt_snap_cli/query/registry.py index 4fcbdf3..08e9208 100644 --- a/src/pt_snap_cli/query/registry.py +++ b/src/pt_snap_cli/query/registry.py @@ -2,6 +2,7 @@ from __future__ import annotations +import copy import warnings from collections.abc import Callable from pathlib import Path @@ -331,9 +332,29 @@ def _load_yaml_templates(template_dir: Path | str | None = None) -> None: warnings.warn(f"Failed to load query template from {yaml_file}: {e}", stacklevel=2) +_packaged: dict[str, tuple[QueryTemplate, QueryTemplate]] = {} + + +def is_packaged_query(name: str) -> bool: + """Only unchanged, actually loaded packaged objects have global semantics. + + A runtime override, even with a built-in name or identical SQL, is custom. + Retain a deep copy too, since QueryTemplate is mutable. + """ + current = get_query(name) + original = _packaged.get(name) + return original is not None and current is original[0] and current == original[1] + + def _load_all_templates() -> None: - """Load all templates from YAML files.""" + """Load all templates from YAML files and record their actual identities.""" _load_yaml_templates() + _packaged.clear() + _packaged.update( + (name, (template, copy.deepcopy(template))) + for name in list_queries() + if (template := get_query(name)) is not None + ) try: diff --git a/src/pt_snap_cli/query/templates/basic/block.yaml b/src/pt_snap_cli/query/templates/basic/block.yaml index e18f1af..195ed74 100644 --- a/src/pt_snap_cli/query/templates/basic/block.yaml +++ b/src/pt_snap_cli/query/templates/basic/block.yaml @@ -120,3 +120,60 @@ queries: type: int - column: freeEventId type: int + - column: lifecycle_id + type: str + units: text + metric_semantics: identifier + scope: captured_range + interpretation_limits: ["Dataset only, optional: generation/device/proved allocation-event or stable negative lifetime token; unproved histories use a separate shard-local identity. Never address or local stack identity."] + - column: identity_status + type: str + units: text + metric_semantics: classification + scope: captured_range + interpretation_limits: ["Dataset only, optional: proved_allocation_event, stable_negative_token, or unproved_shard_local. Unknown histories are not guessed into a global lifetime."] + - column: allocation_source + type: object + metric_semantics: identifier + scope: captured_range + interpretation_limits: ["Dataset only, optional: resolved alloc-action event, owning slice/local stack provenance, canonical full source identity and optionally covered ordered raw frames; NULL for preexisting or missing proof."] + - column: allocation_source_status + type: str + units: text + metric_semantics: classification + scope: captured_range + interpretation_limits: ["Dataset only, optional: resolved, unknown_preexisting, or missing. Missing proof does not discard dynamic bytes or invent an allocation event."] + - column: free_source + type: object + metric_semantics: identifier + scope: captured_range + interpretation_limits: ["Dataset only, optional: proved free_completed event with owning-source provenance, never free_requested; NULL for live/unknown or missing completion proof."] + - column: free_source_status + type: str + units: text + metric_semantics: classification + scope: captured_range + interpretation_limits: ["Dataset only, optional: resolved, live_or_unknown, or missing. No recorded completion alone does not prove a leak."] + - column: observation_slice_index + type: int + metric_semantics: identifier + scope: captured_range + interpretation_limits: ["Dataset only, optional: latest selected shard containing this lifetime observation, not necessarily the terminal dataset shard."] + - column: state_scope + type: str + units: text + metric_semantics: classification + scope: captured_range + interpretation_limits: ["Dataset only, optional: latest_slice_observation. Stored state is not reconstructed at an arbitrary event or at dataset end when the lifetime is absent from the terminal shard."] + - column: terminal_survivor + type: bool + units: flag + metric_semantics: classification + scope: captured_range + interpretation_limits: ["Dataset only, optional: present in the terminal dataset shard with active/pending-free state and no recorded completion. Survival is not a confirmed leak or proof of allocation history."] + - column: category + type: str + units: text + metric_semantics: classification + scope: captured_range + interpretation_limits: ["Dataset only, optional: dynamic, static, or preexisting according to captured lifecycle IDs. Identity/source proof and terminal survival are separate fields."] diff --git a/src/pt_snap_cli/query/templates/basic/event.yaml b/src/pt_snap_cli/query/templates/basic/event.yaml index 9b4b9e0..6f53326 100644 --- a/src/pt_snap_cli/query/templates/basic/event.yaml +++ b/src/pt_snap_cli/query/templates/basic/event.yaml @@ -149,6 +149,18 @@ queries: LIMIT {{ limit }} {% if offset > 0 %}OFFSET {{ offset }}{% endif %} output_schema: + - column: source_stack_id + type: str + interpretation_limits: ["Dataset only with stack_bytes>=0 or recognized ordered-frame declaration: full generation/device/canonical text identity for both layouts, or exact raw ordered-frame identity when covered; source owner/local IDs are provenance, not grouping keys; never merge by shortened text or bare local IDs."] + - column: text_kind + type: str + interpretation_limits: ["Dataset source detail only: captured or missing formatted text. Independent of captured ordered raw frames; text availability never changes raw-array identity."] + - column: frames + type: list + interpretation_limits: ["Dataset only with stack_bytes>=0 or recognized ordered-frame declaration: ordered raw JSON objects preserving duplicates, only for explicit recognized version/schema/per-event coverage; NULL is text-only/uncovered, not reconstructed from formatted text."] + - column: frames_status + type: str + interpretation_limits: ["Dataset only with stack_bytes>=0 or recognized ordered-frame declaration: ordered or text_only; scope.source_coverage covers returned_events separately from result truncation."] - column: id type: int - column: action diff --git a/src/pt_snap_cli/query/templates/business/active_memory_callstack_at_event.yaml b/src/pt_snap_cli/query/templates/business/active_memory_callstack_at_event.yaml index baf0eac..8a9c8c7 100644 --- a/src/pt_snap_cli/query/templates/business/active_memory_callstack_at_event.yaml +++ b/src/pt_snap_cli/query/templates/business/active_memory_callstack_at_event.yaml @@ -230,10 +230,22 @@ queries: ) ORDER BY size_bytes DESC, block_count DESC, callstack ASC output_schema: + - column: source_stack_id + type: str + interpretation_limits: ["Dataset only, always present: full generation/device/canonical text identity for both layouts, or exact raw ordered-frame identity when covered; source owner/local IDs are provenance, not grouping keys; resolve sources before grouping/top_n."] + - column: text_kind + type: str + interpretation_limits: ["Dataset only: captured/missing formatted text or special category; independent of raw ordered-array identity. Representative text is preferred when available."] + - column: frames + type: list + interpretation_limits: ["Dataset only: ordered raw JSON objects preserving duplicates, only for explicit recognized version/schema/per-event coverage; NULL means text-only or uncovered, never reconstructed from text."] + - column: frames_status + type: str + interpretation_limits: ["Dataset only: ordered or text_only; scope.source_coverage describes source/frame coverage before ranking/paging independently of truncated."] - column: stack_id type: str interpretation_limits: - - "Opt-in stack_bytes >= 0 only. Database/device-scoped identity: v1 full-text SHA-256, v2 callstackId, or category identity. Missing, static, preexisting and captured literal labels stay distinct. Never merge by display text." + - "Standalone opt-in stack_bytes >= 0 only; datasets always expose canonical content/ordered-frame identity. Standalone database/device-scoped identity: v1 full-text SHA-256, v2 callstackId, or category identity. Missing, static, preexisting and captured literal labels stay distinct. Never merge by display text." - column: stack_kind type: str interpretation_limits: @@ -264,7 +276,7 @@ queries: interpretation_limits: - "Static and preexisting groups use synthetic labels rather than a captured allocation callstack. Dynamic rows use '[missing callstack]' for empty strings, NULL text or IDs, or missing joined allocation events or callstack records." - "All missing dynamic callstacks merge before top_n ranking; bytes, requested bytes, and block counts are retained. Missing attribution does not imply static or preexisting memory." - - "Nonempty callstacks are grouped by text in v1 and callstackId in v2. Whitespace-only text and a captured literal '[missing callstack]' are not missing and do not merge into the missing group. Display labels alone are not unique group identities." + - "Standalone nonempty callstacks are grouped by text in v1 and callstackId in v2. Datasets resolve local v2 IDs then group canonical full text across owners, or exact raw ordered-frame identity when explicitly covered. Whitespace-only text and a captured literal '[missing callstack]' are not missing and do not merge into the missing group. Display labels alone are not unique group identities." - column: category type: str units: text diff --git a/src/pt_snap_cli/query/templates/business/leak_detection.yaml b/src/pt_snap_cli/query/templates/business/leak_detection.yaml index f9e3004..374b1bf 100644 --- a/src/pt_snap_cli/query/templates/business/leak_detection.yaml +++ b/src/pt_snap_cli/query/templates/business/leak_detection.yaml @@ -51,3 +51,78 @@ queries: interpretation_limits: - "Points to the captured allocation event. This template keeps only allocEventId >= 0, so the uncaptured sentinel -1 is excluded." - "Event IDs are ordering markers, not timestamps." + - column: requestedSize + type: int + units: bytes + metric_semantics: instantaneous_occupancy + scope: dynamic + interpretation_limits: ["Dataset only, optional: user-requested bytes of the terminal candidate's observed block, distinct from its rounded size."] + - column: state + type: int + metric_semantics: classification + scope: dynamic + interpretation_limits: ["Dataset only, optional: terminal-shard observation, active_allocated=1 or active_pending_free=0 for these candidates; not a confirmed leak classification."] + - column: freeEventId + type: int + units: event_id + metric_semantics: ordering_marker + scope: captured_range + sentinel: -1 + interpretation_limits: ["Dataset only, optional: recorded completion ID; these terminal candidates have no recorded free_completed event. -1 is live or unknown, not proof of a leak."] + - column: lifecycle_id + type: str + units: text + metric_semantics: identifier + scope: captured_range + interpretation_limits: ["Dataset only, optional: generation/device/proved allocation-event identity, deduplicated before the final window; never address or local stack identity."] + - column: identity_status + type: str + units: text + metric_semantics: classification + scope: captured_range + interpretation_limits: ["Dataset only, optional: proved_allocation_event for these candidates. Static, preexisting and unproved shard-local histories are excluded."] + - column: allocation_source + type: object + metric_semantics: identifier + scope: captured_range + interpretation_limits: ["Dataset only, optional: resolved alloc-action event, owning slice/local stack provenance, canonical full source identity and optionally covered ordered raw frames."] + - column: allocation_source_status + type: str + units: text + metric_semantics: classification + scope: captured_range + interpretation_limits: ["Dataset only, optional: resolved for these captured dynamic candidates; missing or preexisting allocation proof is not asserted as a leak candidate."] + - column: free_source + type: object + metric_semantics: identifier + scope: captured_range + interpretation_limits: ["Dataset only, optional: completion-source payload; NULL for these candidates without a recorded free_completed event. A free_requested action is not a completed free."] + - column: free_source_status + type: str + units: text + metric_semantics: classification + scope: captured_range + interpretation_limits: ["Dataset only, optional: live_or_unknown for these candidates; absence of completion is not a confirmed leak."] + - column: observation_slice_index + type: int + metric_semantics: identifier + scope: captured_range + interpretation_limits: ["Dataset only, optional: terminal dataset shard containing the candidate observation; earlier-shard locally unfreed observations are not unioned into this result."] + - column: state_scope + type: str + units: text + metric_semantics: classification + scope: captured_range + interpretation_limits: ["Dataset only, optional: latest_slice_observation, which is the terminal shard for these candidates. State is an observation, not a diagnosis of leaking."] + - column: terminal_survivor + type: bool + units: flag + metric_semantics: classification + scope: captured_range + interpretation_limits: ["Dataset only, optional: true for these present, active/pending-free terminal lifetimes without recorded completion; survival is not a confirmed leak."] + - column: category + type: str + units: text + metric_semantics: classification + scope: dynamic + interpretation_limits: ["Dataset only, optional: dynamic for these candidates; static/preexisting categories are excluded independently of output caps."] diff --git a/src/pt_snap_cli/query/templates/statistical/active_blocks_at_event.yaml b/src/pt_snap_cli/query/templates/statistical/active_blocks_at_event.yaml index 227f470..c6bfd94 100644 --- a/src/pt_snap_cli/query/templates/statistical/active_blocks_at_event.yaml +++ b/src/pt_snap_cli/query/templates/statistical/active_blocks_at_event.yaml @@ -75,7 +75,27 @@ queries: {% if order_dir|upper == 'ASC' %}ASC{% else %}DESC{% endif %}, id ASC LIMIT {{ limit|int }} {% if offset > 0 %}OFFSET {{ offset|int }}{% endif %} + interpretation_limits: + - "Dataset point-event queries resolve alloc/free sources in batches before paging; scope.source_coverage describes all filtered blocks and is separate from row truncation. Stored state is a slice observation, not necessarily state at the target event." output_schema: + - column: lifecycle_id + type: str + interpretation_limits: ["Dataset only: generation/device/original alloc-event or stable negative lifetime identity; never address identity."] + - column: state_scope + type: str + interpretation_limits: ["Dataset only: slice_observation; state is not reconstructed at the requested event."] + - column: allocation_source + type: object + interpretation_limits: ["Dataset only: resolved original event, owning slice, full stack identity and optional covered ordered frames; NULL for unknown/preexisting or missing proof."] + - column: free_source + type: object + interpretation_limits: ["Dataset only: completed-free source, never free_requested; NULL for live/unknown or missing proof."] + - column: allocation_source_status + type: str + interpretation_limits: ["Dataset only: resolved, unknown_preexisting or missing; missing dynamic sources retain bytes/category."] + - column: free_source_status + type: str + interpretation_limits: ["Dataset only: resolved, live_or_unknown or missing; no recorded free does not prove a leak."] - column: id type: int - column: address diff --git a/src/pt_snap_cli/query/templates/statistical/allocator_gap.yaml b/src/pt_snap_cli/query/templates/statistical/allocator_gap.yaml index a662b8e..7bc0cdf 100644 --- a/src/pt_snap_cli/query/templates/statistical/allocator_gap.yaml +++ b/src/pt_snap_cli/query/templates/statistical/allocator_gap.yaml @@ -4,8 +4,9 @@ queries: description: Compare allocated, active, and reserved peak events and same-event gaps devices: - all - semantics_version: 1 + semantics_version: 2 interpretation_limits: + - "Only real trace events (id >= 0) select peaks and same-event counters, matching memory_peak semantics version 2." - "Gaps are computed from counters at the same peak event. A reserved-active gap is not itself fragmentation." - "Allocated, active, and reserved may peak at different events; use the same-event flags before comparing across metrics." - "Event IDs are ordering markers, not elapsed time." @@ -24,7 +25,7 @@ queries: WITH filtered AS ( SELECT id, allocated, active, reserved FROM {{ device_trace_table }} - WHERE 1=1 + WHERE id >= 0 {% if start_id is not none %}AND id >= {{ start_id|int }}{% endif %} {% if end_id is not none %}AND id <= {{ end_id|int }}{% endif %} ), peaks AS ( diff --git a/src/pt_snap_cli/query/templates/statistical/callstack_analysis.yaml b/src/pt_snap_cli/query/templates/statistical/callstack_analysis.yaml index 7cdb3a8..67512a5 100644 --- a/src/pt_snap_cli/query/templates/statistical/callstack_analysis.yaml +++ b/src/pt_snap_cli/query/templates/statistical/callstack_analysis.yaml @@ -1,7 +1,12 @@ version: "1.0" queries: callstack_analysis: - description: Analyze memory allocation by callstack + description: Analyze all real stack-bearing trace actions by full callstack identity + semantics_version: 2 + interpretation_limits: + - "alloc_count is a retained legacy field name: it counts ALL real stack-bearing trace actions, not allocation-only events. v1 requires non-NULL text; v2 requires a non-NULL stack reference and retains absent/NULL joined text as a separate missing group. Empty text counts. Negative boundaries are excluded." + - "Statistics group full captured text in both standalone layouts; dataset grouping uses canonical text or explicitly covered ordered raw arrays before thresholds and ranking." + - "avg_size is total_size/alloc_count, not an average of shard averages; size sums include free/segment/workspace actions and are not live occupancy." devices: - all - "0" @@ -11,7 +16,7 @@ queries: type: int default: 5 required: false - description: Minimum allocation count to include + description: Minimum real stack-bearing action count (legacy alloc_count) to include min_size: type: int default: 0 @@ -31,7 +36,7 @@ queries: AVG(size) as avg_size, MAX(size) as max_size FROM {{ device_trace_table }} - WHERE callstack IS NOT NULL + WHERE id >= 0 AND callstack IS NOT NULL GROUP BY callstack HAVING alloc_count >= {{ min_count }} AND total_size >= {{ min_size }} @@ -40,25 +45,17 @@ queries: v2: | SELECT COALESCE(c.callstack, '[missing callstack]') AS callstack, - grouped.alloc_count AS alloc_count, - grouped.total_size AS total_size, - grouped.avg_size AS avg_size, - grouped.max_size AS max_size - FROM ( - SELECT - callstackId, - COUNT(*) as alloc_count, - SUM(size) as total_size, - AVG(size) as avg_size, - MAX(size) as max_size - FROM {{ device_trace_table }} - WHERE callstackId IS NOT NULL - GROUP BY callstackId - HAVING alloc_count >= {{ min_count }} - AND total_size >= {{ min_size }} - ) AS grouped - LEFT JOIN callstack c ON c.id = grouped.callstackId - ORDER BY total_size DESC, callstack ASC + COUNT(*) AS alloc_count, + SUM(t.size) AS total_size, + AVG(t.size) AS avg_size, + MAX(t.size) AS max_size + FROM {{ device_trace_table }} t + LEFT JOIN callstack c ON c.id = t.callstackId + WHERE t.id >= 0 AND t.callstackId IS NOT NULL + GROUP BY c.callstack + HAVING alloc_count >= {{ min_count }} + AND total_size >= {{ min_size }} + ORDER BY total_size DESC, callstack ASC, c.callstack IS NULL ASC LIMIT {{ limit|int }} output_schema: - column: callstack @@ -71,3 +68,33 @@ queries: type: float - column: max_size type: int + - column: source_stack_id + type: str + units: text + metric_semantics: identifier + scope: captured_range + interpretation_limits: ["Dataset only, optional: generation/device/canonical full-text or explicitly covered ordered raw-array identity, before global thresholds/ranking. Empty non-NULL text is a separate statistical group from a missing reference. Never group by local IDs, display labels or text_kind."] + - column: stack_kind + type: str + units: text + metric_semantics: classification + scope: captured_range + interpretation_limits: ["Dataset only, optional: shared source availability, captured or missing; explicitly covered raw arrays (including empty arrays) are captured even without formatted text. Empty non-NULL text still counts statistically but does not prove captured attribution."] + - column: text_kind + type: str + units: text + metric_semantics: classification + scope: captured_range + interpretation_limits: ["Dataset only, optional: captured for nonempty formatted text, otherwise missing. Provenance only: independent of raw-array identity/availability and never a grouping key."] + - column: stack_event_id + type: int + units: event_id + metric_semantics: ordering_marker + scope: captured_range + interpretation_limits: ["Dataset only, optional: representative real source event for retrieval on the same generation/device; prefers nonempty captured text when available, otherwise may point to empty/NULL text. Not the group identity, not necessarily an allocation action, and not a timestamp."] + - column: frames_status + type: str + units: text + metric_semantics: classification + scope: captured_range + interpretation_limits: ["Dataset only, optional: ordered only for the recognized version/schema and valid per-event raw-array coverage, otherwise text_only. Retrieve actual arrays with event(id=stack_event_id); statistics do not reconstruct or emit frame arrays."] diff --git a/src/pt_snap_cli/query/templates/statistical/memory_peak.yaml b/src/pt_snap_cli/query/templates/statistical/memory_peak.yaml index 5d576ea..3e4a2ec 100644 --- a/src/pt_snap_cli/query/templates/statistical/memory_peak.yaml +++ b/src/pt_snap_cli/query/templates/statistical/memory_peak.yaml @@ -4,8 +4,9 @@ queries: description: Get peak memory metrics from trace events with corresponding event IDs devices: - all - semantics_version: 1 + semantics_version: 2 interpretation_limits: + - "Only real trace events (id >= 0) participate in standalone databases and datasets; synthetic boundary occupancy cannot be a peak." - "Each peak is the maximum of that counter in the filtered event range; ties resolve to the earliest event ID." - "Peaks of allocated, active, and reserved may occur at different events; do not subtract them as if they were simultaneous." - "Event IDs are ordering markers, not elapsed time." @@ -30,7 +31,7 @@ queries: WITH filtered AS ( SELECT id, allocated, active, reserved FROM {{ device_trace_table }} - WHERE 1=1 + WHERE id >= 0 {% if start_id is not none %}AND id >= {{ start_id|int }}{% endif %} {% if end_id is not none %}AND id <= {{ end_id|int }}{% endif %} ), peaks AS ( diff --git a/src/pt_snap_cli/snapshot/PROVENANCE.md b/src/pt_snap_cli/snapshot/PROVENANCE.md index 75f3fd7..a84aa16 100644 --- a/src/pt_snap_cli/snapshot/PROVENANCE.md +++ b/src/pt_snap_cli/snapshot/PROVENANCE.md @@ -228,3 +228,61 @@ discarded side-effect return values explicit. Database/slice adaptor consumers annotate the existing per-device trace-list and segment-list format assumptions. No serialization, validation, replay, or database layout behavior changes. Audited upstream source mappings and license terms remain unchanged. + +## 2026-10-05 — Continuous native shard replay (issue #199) + +Interface facts were independently read through real GitHub GETs at +Ascend/msinsight revision `101f65b877a267ffd5f66ea3834706057ba243e5`: +[resumable replay](https://api.github.com/repos/Ascend/msinsight/git/blobs/1956490d46ccf11309d10853ea76175d0713934e) +and [boundary/registry interfaces](https://api.github.com/repos/Ascend/msinsight/git/blobs/95612509ef990ac6b7034919dae4938cc6acf3ad). +The fixed tree confirms both blob identities. These sources carry Copyright (c) +2026 Huawei Technologies Co.,Ltd. and Mulan PSL v2; the docs' CC BY 4.0 license +is separate. No upstream implementation is copied into the package. + +Local original implementation changes `simulate/simulate.py` to reuse the existing +full replay loop for position-based pauses (deliberately not an ID threshold), +adds `tools/adaptors/sharded_replay.py` for native-v2 multi-writer staging, +original-object lifecycle tracking, negative left-boundary rows, transactional +batch ID/free_completed/stack-source backfill and final validation, and adds the +shared `core/sharded_replay_service.py` producer boundary. Unknown allocations +keep device-local negative identities without upstream compaction. All writers +close before finalization; no incremental readiness, compatibility manifest, +cache/publication, CLI changes, or structured-frame extension is introduced. +`tools/adaptors/snapshot2db.py` and `database/snapshot_db.py` close connections on +constructor/commit failure while preserving the original single-DB data contract. +Synthetic failure/equivalence tests and bilingual docs are locally authored; +reviewed fixtures are unchanged. Existing source mappings and MIT relicensing +records are retained. GUI interoperability and streaming/RSS claims are not made. + +## 2026-10-05 — Read-only dataset focus and addressing (issue #200) + +Local original reader changes in `core/dataset_resolver.py`, focus/overview/query +services and CLI/API adapters reuse the committed P0 compatibility-v1 validator. +Manifest/device/slice fields, real-event range routing and completion facts were +independently read via a fixed GitHub GET at Ascend/msinsight revision +`101f65b877a267ffd5f66ea3834706057ba243e5`, source +[manifest/routing interfaces](https://api.github.com/repos/Ascend/msinsight/git/blobs/8746d17db5a8e66b2ed0843962593c12851414e0). +The source carries Copyright (c) 2026 Huawei Technologies Co.,Ltd. and Mulan PSL +v2. No C++ implementation was copied; Python logic, synthetic SQLite regressions +and bilingual reader documentation are locally authored. Complete-readiness and +canonical artifact rules remain stricter than upstream building inspection. + +No allocator, importer, compatibility exporter, native-v2 staging protocol, +fixture, license or retained source mapping is changed. The reader reports text +stacks without promoting structured frames, refuses silent cross-slice template +execution, and never loads source pickle or repairs external databases. Context +LRU generation uses manifest/member content identity; real overview bounds exclude +negative boundary rows. No GUI, wheel/profile or performance acceptance is claimed. + +## 2026-10-05 — Dataset read-only WAL preflight correction (issue #200) + +Local original correction in `core/dataset_resolver.py`: verify every declared +member's canonical non-symlink path, WAL/journal/SHM sidecars (including dangling +aliases) and SQLite file-header read/write versions before any SQLite connection. +Checkpointed persistent-WAL databases are rejected even without existing sidecars, +because `mode=ro` can recreate WAL/SHM files. Synthetic real-SQLite regressions +record unchanged member/manifest hashes, inventory and no connection opens on +rejection across resolver, focus, CLI and API. No repair, checkpoint, journal-mode +change, artifact deletion or standalone-DB policy change is introduced. This is +not concurrent-producer locking. Existing attribution, source mappings, fixtures +and license records are unchanged; no live upstream artifact acceptance is claimed. diff --git a/src/pt_snap_cli/snapshot/simulate/simulate.py b/src/pt_snap_cli/snapshot/simulate/simulate.py index 7e8d942..524bfba 100644 --- a/src/pt_snap_cli/snapshot/simulate/simulate.py +++ b/src/pt_snap_cli/snapshot/simulate/simulate.py @@ -154,14 +154,27 @@ def unregister_allocator_hooker(self, hooker_id: int): self.simulated_allocator.unregister_hooker(hooker_id) def replay(self) -> bool: + """Undo the entire remaining trace through the resumable replay path.""" + return self.replay_until(0) + + def replay_until(self, remaining_events: int) -> bool: + """Undo until exactly ``remaining_events`` chronological entries remain. + + The endpoint is a list position, not an event ID (native IDs may be sparse). + Undoing positions [start, end] leaves the state immediately BEFORE start. + Pre hooks observe the state AFTER the event; post hooks observe BEFORE it. + A pre/allocator failure leaves the event pending, while a post-hook failure + occurs after it was consumed, exactly as for full replay. """ - 开始仿真回放内存事件 - """ + if type(remaining_events) is not int or not ( + 0 <= remaining_events <= len(self.device_snapshot.trace_entries) + ): + raise ValueError("remaining_events must be an integer within the remaining trace") # 倒序遍历 total_size = len(self.device_snapshot.trace_entries) self._replay_logger.info(f"Replaying {total_size} entries in snapshot...") progress_update_point = [0.25, 0.5, 0.75] - while self.device_snapshot.trace_entries: + while len(self.device_snapshot.trace_entries) > remaining_events: for hooker in self.hookers.values(): if hooker and not hooker.pre_undo_event( self.device_snapshot.trace_entries[-1], self.device_snapshot @@ -191,5 +204,9 @@ def replay(self) -> bool: "An interruption occurred during the replay of the single event post hook." ) return False - self._replay_logger.info("All events have been successfully replayed.") + self._replay_logger.info( + "All events have been successfully replayed." + if not remaining_events + else f"Replay paused with {remaining_events} entries remaining." + ) return True diff --git a/src/pt_snap_cli/snapshot/tools/adaptors/database/snapshot_db.py b/src/pt_snap_cli/snapshot/tools/adaptors/database/snapshot_db.py index 6fd43b2..ef71010 100644 --- a/src/pt_snap_cli/snapshot/tools/adaptors/database/snapshot_db.py +++ b/src/pt_snap_cli/snapshot/tools/adaptors/database/snapshot_db.py @@ -64,13 +64,18 @@ class SnapshotDb(SqliteDB): CALLSTACK_TABLE_NAME = "callstack" def __init__(self, path: str): - super().__init__(path, auto_create=True, with_dictionary_table=True) - # Import builds a replaceable database, so favor write throughput over crash recovery. - self.conn.execute("PRAGMA journal_mode = MEMORY") - self.conn.execute("PRAGMA synchronous = OFF") - self.conn.execute("PRAGMA cache_size = -65536") - # 清理旧版本表格 - self._clear_old_tables() + try: + super().__init__(path, auto_create=True, with_dictionary_table=True) + # Import builds a replaceable database, so favor write throughput over crash recovery. + self.conn.execute("PRAGMA journal_mode = MEMORY") + self.conn.execute("PRAGMA synchronous = OFF") + self.conn.execute("PRAGMA cache_size = -65536") + # 清理旧版本表格 + self._clear_old_tables() + except BaseException: + if hasattr(self, "conn"): + self.conn.close() + raise def create_trace_entry_table(self, device: int = 0): self.create_table( diff --git a/src/pt_snap_cli/snapshot/tools/adaptors/sharded_replay.py b/src/pt_snap_cli/snapshot/tools/adaptors/sharded_replay.py new file mode 100644 index 0000000..1d2d130 --- /dev/null +++ b/src/pt_snap_cli/snapshot/tools/adaptors/sharded_replay.py @@ -0,0 +1,278 @@ +"""Internal native-v2 shard construction; no compatibility export or publication. + +All writers close before finalization. Only a successful return makes identities +stable. Partial files in the caller's private staging directory are NOT ready. +""" + +from __future__ import annotations + +import sqlite3 +from contextlib import closing +from dataclasses import dataclass, field +from pathlib import Path +from typing import Any + +from ...base import Block, BlockState, DeviceSnapshot, TraceEntry +from ...simulate import AllocatorHooker, SimulateDeviceSnapshot, SimulateHooker +from .database import CallstackInterner, event2record +from .snapshot2db import SnapshotDbHandler + + +@dataclass(frozen=True) +class ReplaySlice: + device: int + index: int + start_position: int + end_position: int + start_event_id: int + end_event_id: int + file: Path + + +@dataclass +class _Lifetime: + # Retain the ORIGINAL object: copied allocator hook payloads and address reuse + # cannot alias it, and Python cannot recycle its object identity during replay. + block: Block + token: int + stream: int + alloc_stack: str | None = None + free_stack: str | None = None + + @property + def final_id(self) -> int: + return self.token if self.block.alloc_event_idx is None else self.block.alloc_event_idx + + def signature(self) -> tuple[int, int, int, int, int]: + block = self.block + return ( + block.address, + block.size, + block.requested_size, + -1 if block.alloc_event_idx is None else block.alloc_event_idx, + -1 if block.free_event_idx is None else block.free_event_idx, + ) + + +@dataclass +class BlockRegistry: + """One snapshot/device-local registry; provisional IDs never escape staging.""" + + lifetimes: dict[int, _Lifetime] = field(default_factory=dict) + members: dict[Path, dict[int, _Lifetime]] = field(default_factory=dict) + + def observe(self, block: Block, stream: int) -> _Lifetime: + key = id(block) + lifetime = self.lifetimes.get(key) + if lifetime is None: + lifetime = _Lifetime(block, -len(self.lifetimes) - 1, stream) + self.lifetimes[key] = lifetime + elif lifetime.stream != stream: + raise RuntimeError("Block changed stream during its lifetime") + return lifetime + + def record(self, path: Path, block: Block, stream: int) -> dict[str, Any] | None: + lifetime = self.observe(block, stream) + members = self.members.setdefault(path, {}) + if lifetime.token in members: + return None + members[lifetime.token] = lifetime + return { + "id": lifetime.token, + "address": block.address, + "size": block.size, + "requestedSize": block.requested_size, + "state": block.state, + "allocEventId": -1, + "freeEventId": -1 if block.free_event_idx is None else block.free_event_idx, + } + + def finalize(self) -> None: + """Batch patches by closed DB, including free_completed and stack sources. + + Unknown/preexisting allocations keep their unique negative token; no + compaction is necessary or allowed to merge unrelated objects. + """ + for path, members in self.members.items(): + device = int(path.parent.name.removeprefix("device_")) + with closing(sqlite3.connect(path.as_uri() + "?mode=rw", uri=True)) as conn, conn: + changes = conn.total_changes + conn.executemany( + f'UPDATE "block_{device}" SET id=?, allocEventId=?, freeEventId=? WHERE id=?', + [ + (item.final_id, *item.signature()[-2:], token) + for token, item in members.items() + ], + ) + if conn.total_changes - changes != len(members): + raise RuntimeError("Missing provisional block during finalization") + conn.execute( + "CREATE TABLE pt_snap_block_reference (blockId INTEGER PRIMARY KEY, " + "stream INTEGER NOT NULL, allocCallstack TEXT, freeCallstack TEXT)" + ) + conn.executemany( + "INSERT INTO pt_snap_block_reference VALUES (?, ?, ?, ?)", + [ + (item.final_id, item.stream, item.alloc_stack, item.free_stack) + for item in members.values() + ], + ) + + def validate(self, item: ReplaySlice, expected_ids: list[int]) -> None: + with closing(sqlite3.connect(item.file.as_uri() + "?mode=ro", uri=True)) as conn: + if conn.execute("PRAGMA quick_check").fetchall() != [("ok",)]: + raise RuntimeError("Invalid finalized SQLite shard") + ids = [ + row[0] + for row in conn.execute( + f"SELECT id FROM trace_entry_{item.device} WHERE id>=0 ORDER BY id" + ) + ] + if ids != expected_ids: + raise RuntimeError("Shard lost or changed real events") + rows = conn.execute( + f"SELECT id,address,size,requestedSize,allocEventId,freeEventId FROM block_{item.device}" + ).fetchall() + expected = { + (lifetime.final_id, *lifetime.signature()) + for lifetime in self.members[item.file].values() + } + if len(rows) != len(expected) or set(rows) != expected: + raise RuntimeError("Incomplete block identity/lifetime finalization") + references = conn.execute("SELECT * FROM pt_snap_block_reference").fetchall() + if set(references) != { + (lifetime.final_id, lifetime.stream, lifetime.alloc_stack, lifetime.free_stack) + for lifetime in self.members[item.file].values() + }: + raise RuntimeError("Incomplete cross-slice callstack references") + + +class _SliceHook(SimulateHooker, AllocatorHooker): + def __init__(self, item: ReplaySlice, registry: BlockRegistry): + self.item = item + self.registry = registry + self.writer = SnapshotDbHandler(str(item.file), [item.device]) + self.stacks = CallstackInterner() + self.current_event: TraceEntry | None = None + + def pre_undo_event(self, wait4undo_event: TraceEntry, current_snapshot: DeviceSnapshot) -> bool: + event, snapshot = wait4undo_event, current_snapshot + self.current_event = event + self.writer.insert_event( + event2record( + event, + snapshot.total_allocated, + snapshot.total_activated, + snapshot.total_reserved, + callstacks=self.stacks, + ), + snapshot.device, + ) + return True + + def post_undo_event( + self, already_undo_event: TraceEntry, current_snapshot: DeviceSnapshot + ) -> bool: + return True + + def post_replay_alloc_block(self, allocated_block: Block, current_snapshot: DeviceSnapshot): + assert self.current_event is not None + lifetime = self.registry.observe(allocated_block, self.current_event.stream) + lifetime.free_stack = self.current_event.get_callstack() + + def pre_replay_free_block(self, wait4free_block: Block, current_snapshot: DeviceSnapshot): + # This hook receives the original block before detach; the post hook is + # a shallow copy and MUST NOT be used as an object-identity registry key. + assert self.current_event is not None + lifetime = self.registry.observe(wait4free_block, self.current_event.stream) + lifetime.alloc_stack = self.current_event.get_callstack() + self._write_block(wait4free_block, lifetime.stream, current_snapshot.device) + + def _write_block(self, block: Block, stream: int, device: int): + record = self.registry.record(self.item.file, block, stream) + if record is not None: + self.writer.insert_block(record, device) + + def dump_left_boundary(self, snapshot: DeviceSnapshot): + # Called AFTER undoing the first event in this window. Segment rows and + # block states describe the instant BEFORE that first event, not after. + for index, segment in enumerate(snapshot.segments, start=1): + for block in segment.blocks: + if block.state != BlockState.INACTIVE: + self._write_block(block, segment.stream, snapshot.device) + boundary = TraceEntry( + idx=-index, + action="segment_map" if segment.is_expandable else "segment_alloc", + addr=segment.address, + size=segment.total_size, + stream=segment.stream, + frames=segment.frames, + _raw_frames=segment._raw_frames, + ) + self.writer.insert_event( + event2record( + boundary, + snapshot.total_allocated, + snapshot.total_activated, + snapshot.total_reserved, + callstacks=self.stacks, + ), + snapshot.device, + ) + + +def build_shards( + data: dict[str, Any], directory: Path, events_per_slice: int, devices: tuple[int, ...] +) -> tuple[ReplaySlice, ...]: + """Build inside a caller-owned private EMPTY directory; never publish readiness. + + Core validates options and owns staging. Native v2 preserves OOM/workspace; + this is NOT the compatibility-v1 dataset validated by dataset_contract. + """ + results: list[ReplaySlice] = [] + for device in devices: + simulator = SimulateDeviceSnapshot(data, device, _raw_frames=True) + events = simulator.device_snapshot.trace_entries + event_ids = [event.idx for event in events] + if any(type(value) is not int or value < 0 for value in event_ids): + raise ValueError("Real event IDs must be nonnegative integers") + ids = [int(value) for value in event_ids if value is not None] + if ids != sorted(set(ids)): + raise ValueError("Real event IDs must be unique and chronological") + registry = BlockRegistry() + device_dir = directory / f"device_{device}" + device_dir.mkdir() + slices: list[ReplaySlice] = [] + for index in reversed(range((len(events) + events_per_slice - 1) // events_per_slice)): + start = index * events_per_slice + end = min(start + events_per_slice, len(ids)) - 1 + item = ReplaySlice( + device, + index, + start, + end, + ids[start], + ids[end], + device_dir / f"slice_{index:05d}.db", + ) + registry.members[item.file] = {} + hook = _SliceHook(item, registry) + hook_id = simulator.register_hooker(hook) + allocator_id = simulator.register_allocator_hooker(hook) + try: + if not simulator.replay_until(start): + raise RuntimeError(f"Replay failed for device {device}, slice {index}") + hook.dump_left_boundary(simulator.device_snapshot) + hook.writer.flush(device) + hook.writer.insert_callstacks(hook.stacks.records()) + hook.writer.close() + finally: + simulator.unregister_hooker(hook_id) + simulator.unregister_allocator_hooker(allocator_id) + hook.writer.close(commit=False) + slices.append(item) + registry.finalize() + for item in slices: + registry.validate(item, ids[item.start_position : item.end_position + 1]) + results.extend(reversed(slices)) + return tuple(results) diff --git a/src/pt_snap_cli/snapshot/tools/adaptors/snapshot2db.py b/src/pt_snap_cli/snapshot/tools/adaptors/snapshot2db.py index f9b84c5..282280a 100644 --- a/src/pt_snap_cli/snapshot/tools/adaptors/snapshot2db.py +++ b/src/pt_snap_cli/snapshot/tools/adaptors/snapshot2db.py @@ -21,18 +21,23 @@ class SnapshotDbHandler: def __init__( self, db_path: str, devices: list[int], insert_cache_size: int = DEFAULT_INSERT_CACHE_SIZE ): - self._closed = False + self._closed = True self.db_path = db_path self.db = SnapshotDb(db_path) + self._closed = False self._device_event_cache = {} self._device_block_cache = {} self._insert_cache_size = insert_cache_size - self.db.create_callstack_table() - for device in devices: - self._device_block_cache[device] = [] - self._device_event_cache[device] = [] - self.db.create_trace_entry_table(device) - self.db.create_block_table(device) + try: + self.db.create_callstack_table() + for device in devices: + self._device_block_cache[device] = [] + self._device_event_cache[device] = [] + self.db.create_trace_entry_table(device) + self.db.create_block_table(device) + except BaseException: + self.close(commit=False) + raise def insert_event(self, event_record: dict[str, Any], device: int = 0): if device not in self._device_event_cache: @@ -83,12 +88,14 @@ def _do_insert_blocks(self, device: int = 0): def close(self, *, commit: bool = True): if getattr(self, "_closed", True): return - if commit: - self.db.conn.commit() - else: - self.db.conn.rollback() - self.db.conn.close() - self._closed = True + try: + if commit: + self.db.conn.commit() + else: + self.db.conn.rollback() + finally: + self.db.conn.close() + self._closed = True def __del__(self): self.close() diff --git a/tests/core/test_dataset_attribution.py b/tests/core/test_dataset_attribution.py new file mode 100644 index 0000000..e797a9f --- /dev/null +++ b/tests/core/test_dataset_attribution.py @@ -0,0 +1,699 @@ +"""Real SQLite contracts for point-event attribution; no original producer execution.""" + +import dataclasses +import json +import sqlite3 +import time +from contextlib import closing +from pathlib import Path + +import pytest + +from pt_snap_cli.api import SnapshotAnalyzer +from pt_snap_cli.core.context_cache import ContextCache +from pt_snap_cli.core.dataset_resolver import DatasetResolver +from pt_snap_cli.core.dataset_sources import SOURCE_BATCH_SIZE, DatasetSourceResolver, QueryBudget +from pt_snap_cli.core.errors import InvalidParameterError, QueryTimeoutError +from pt_snap_cli.core.import_service import ImportService +from pt_snap_cli.core.query_service import QueryService +from pt_snap_cli.core.report_service import ReportService +from tests.core.test_dataset_import import source_options +from tests.test_dataset_focus import hashes, make_dataset + + +@pytest.fixture(autouse=True) +def isolate(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + monkeypatch.setenv("HOME", str(tmp_path)) + monkeypatch.setattr(Path, "home", lambda: tmp_path) + monkeypatch.delenv("PT_SNAP_DB_PATH", raising=False) + monkeypatch.delenv("PT_SNAP_QUERY_TIMEOUT", raising=False) + + +def case(tmp_path, devices=(0, 2)): + root = make_dataset(tmp_path / "no-extensions", devices=devices, slices=8) + blocks = [ + (0, 100, 90, 85, 0, 0, 10), + (1, 200, 60, 55, 1, 1, 14), + (3, 300, 40, 35, 1, 3, -1), + (4, 400, 30, 25, 1, 4, -1), + (5, 500, 25, 20, 1, 5, -1), + (6, 600, 20, 15, 1, 6, -1), + (7, 700, 15, 10, 1, 7, -1), + (11, 100, 80, 75, 1, 11, -1), + (-20, 800, 10, 5, 1, -1, 12), + (-21, 900, 5, 5, 1, -1, -1), + ] + for device in devices: + for index in range(8): + with ( + closing(sqlite3.connect(root / f"device_{device}/slice_{index:05d}.db")) as conn, + conn, + ): + conn.executemany(f"INSERT INTO block_{device} VALUES (?,?,?,?,?,?,?)", blocks) + for event in range(index * 2, index * 2 + 2): + text = { + 0: "shared:" + "长" * 100, + 1: "shared:" + "长" * 100, + 3: "third", + 4: "[missing callstack]", + 5: " \n", + 6: "", + 7: None, + 11: "reused", + }.get(event, f"free:{event}") + action = ( + 4 + if event in (0, 1, 3, 4, 5, 6, 7, 11) + else 6 if event in (10, 12, 14) else 5 if event == 2 else 7 + ) + active = sum( + b[2] + for b in blocks + if (b[5] == -1 or b[5] <= event) and (b[6] == -1 or b[6] > event) + ) + conn.execute( + f"UPDATE trace_entry_{device} SET action=?, active=?, allocated=?, reserved=1000, callstack=? WHERE id=?", + (action, active, active - (90 if 2 <= event < 10 else 0), text, event), + ) + return root + + +def standalone(root, tmp_path, device=0): + target = tmp_path / f"standalone-{device}.db" + with closing(sqlite3.connect(target)) as out, out: + out.executescript( + 'CREATE TABLE dictionary ("table" TEXT,"column" TEXT,"key" TEXT,"value" TEXT);' + f"CREATE TABLE trace_entry_{device} (id INTEGER PRIMARY KEY, action INTEGER,address INTEGER,size INTEGER,stream INTEGER,allocated INTEGER,active INTEGER,reserved INTEGER,callstack TEXT);" + f"CREATE TABLE block_{device} (id INTEGER PRIMARY KEY,address INTEGER,size INTEGER,requestedSize INTEGER,state INTEGER,allocEventId INTEGER,freeEventId INTEGER);" + ) + for index in range(8): + with closing( + sqlite3.connect( + (root / f"device_{device}/slice_{index:05d}.db").as_uri() + + "?mode=ro&immutable=1", + uri=True, + ) + ) as conn: + rows = conn.execute(f"SELECT * FROM trace_entry_{device} WHERE id>=0").fetchall() + out.executemany( + f"INSERT INTO trace_entry_{device} VALUES (?,?,?,?,?,?,?,?,?)", rows + ) + if index == 0: + out.executemany( + "INSERT INTO dictionary VALUES (?,?,?,?)", + conn.execute("SELECT * FROM dictionary").fetchall(), + ) + out.executemany( + f"INSERT INTO block_{device} VALUES (?,?,?,?,?,?,?)", + conn.execute(f"SELECT * FROM block_{device}").fetchall(), + ) + return target + + +def normalized_groups(rows): + return sorted( + ( + r["callstack"], + r["category"], + r["size_bytes"], + r["requested_bytes"], + r["block_count"], + r["percent_of_active_blocks"], + ) + for r in rows + ) + + +@pytest.mark.parametrize("event", [0, 2, 8, 10, 11, 12, 14, 15]) +def test_original_noextensions_matches_single_db_after_event_and_completed_free(tmp_path, event): + root = case(tmp_path) + single = standalone(root, tmp_path) + before = hashes(root) + with SnapshotAnalyzer(root) as dataset, SnapshotAnalyzer(single) as reference: + blocks = dataset.execute_query("active_blocks_at_event", {"event_id": event}) + local = reference.execute_query("active_blocks_at_event", {"event_id": event}) + assert {b["id"] for b in blocks["rows"]} == {b["id"] for b in local["rows"]} + counter = dataset.execute_query("event", {"id": event})["rows"][0]["active"] + assert sum(b["size"] for b in blocks["rows"]) == counter + params = {"event_id": event, "top_n": -1} + grouped = dataset.execute_query("active_memory_callstack_at_event", params) + assert normalized_groups(grouped["rows"]) == normalized_groups( + reference.execute_query("active_memory_callstack_at_event", params)["rows"] + ) + coverage = grouped["scope"]["source_coverage"] + assert coverage["active_bytes"] == counter + assert coverage["allocation_source_complete"] + assert not coverage["ordered_frames_complete"] + assert grouped["scope"]["boundary_events_included"] is False + assert ( + blocks["scope"]["source_coverage"]["free_events_resolved"] + == blocks["scope"]["source_coverage"]["free_references"] + ) + assert all(b["state_scope"] == "slice_observation" for b in blocks["rows"]) + if event == 2: + pending = next(b for b in blocks["rows"] if b["id"] == 0) + assert pending["free_source"]["event"]["id"] == 10 # Not free_requested=2. + assert pending["free_source"]["event"]["action"] == 6 + if event == 11: + reused = next(b for b in blocks["rows"] if b["address"] == 100) + assert reused["id"] == 11 and reused["allocation_source"]["event"]["id"] == 11 + assert before == hashes(root) + assert not list(root.rglob("*-wal")) + + +def test_source_before_top_n_bytes_denominator_specials_missing_literal_and_summary(tmp_path): + root = case(tmp_path) + with SnapshotAnalyzer(root) as analyzer: + result = analyzer.execute_query( + "active_memory_callstack_at_event", {"event_id": 8, "top_n": 1, "stack_bytes": 5} + ) + assert [r["size_bytes"] for r in result["rows"]] == [150, 10, 5] + assert result["rows"][0]["percent_of_active_blocks"] == round(15000 / 165, 4) + assert result["rows"][0]["stack_event_id"] == 0 + assert result["rows"][0]["stack_truncated"] + assert result["has_more"] and result["truncated"] + assert result["scope"]["source_coverage"]["active_bytes"] == 295 + assert result["scope"]["source_coverage"]["allocation_source_complete"] + smaller = analyzer.execute_query( + "active_memory_callstack_at_event", + {"event_id": 8, "top_n": 1}, + max_rows=1, + exact_total=True, + ) + assert smaller["total"] == 7 and smaller["returned"] == 1 + assert ( + smaller["rows"][0]["percent_of_active_blocks"] + == result["rows"][0]["percent_of_active_blocks"] + ) + complete = analyzer.execute_query( + "active_memory_callstack_at_event", {"event_id": 8, "top_n": -1} + ) + labels = [r for r in complete["rows"] if r["callstack"] == "[missing callstack]"] + assert len(labels) == 2 + assert {r["stack_kind"] for r in labels} == {"missing", "captured"} + assert len({r["stack_id"] for r in labels}) == 2 + assert next(r for r in complete["rows"] if r["stack_kind"] == "missing")["size_bytes"] == 35 + assert ( + next(r for r in complete["rows"] if r["callstack"] == " \n")["stack_kind"] + == "captured" + ) + exclude = analyzer.execute_query( + "active_memory_callstack_at_event", + {"event_id": 8, "include_static": False, "min_size": 40, "top_n": -1}, + exact_total=True, + ) + assert sum(r["size_bytes"] for r in exclude["rows"]) == 190 + assert sum(r["percent_of_active_blocks"] for r in exclude["rows"]) == pytest.approx( + 100, abs=0.0001 + ) + + +def test_devices_address_and_negative_lifecycle_identity_are_independent(tmp_path): + root = case(tmp_path) + with SnapshotAnalyzer(root) as analyzer: + one = analyzer.execute_query("active_blocks_at_event", {"event_id": 8}, device_id=0)["rows"] + two = analyzer.execute_query("active_blocks_at_event", {"event_id": 8}, device_id=2)["rows"] + assert not {b["lifecycle_id"] for b in one} & {b["lifecycle_id"] for b in two} + unknown = {b["category"]: b for b in one if b["allocEventId"] == -1} + assert set(unknown) == {"static", "preexisting_live_at_event"} + assert unknown["static"]["free_source_status"] == "live_or_unknown" + assert all(b["allocation_source_status"] == "unknown_preexisting" for b in unknown.values()) + old = next(b for b in one if b["id"] == 0) + new = next( + b + for b in analyzer.execute_query("active_blocks_at_event", {"event_id": 11})["rows"] + if b["id"] == 11 + ) + assert old["address"] == new["address"] and old["lifecycle_id"] != new["lifecycle_id"] + + +def test_missing_dynamic_action_is_not_static_or_row_truncation(tmp_path): + root = case(tmp_path) + with closing(sqlite3.connect(root / "device_0/slice_00000.db")) as conn, conn: + conn.execute("UPDATE trace_entry_0 SET action=2 WHERE id=0") + with SnapshotAnalyzer(root) as analyzer: + result = analyzer.execute_query( + "active_memory_callstack_at_event", {"event_id": 8, "top_n": -1} + ) + assert not result["truncated"] + assert not result["scope"]["source_coverage"]["allocation_source_complete"] + assert sum(r["size_bytes"] for r in result["rows"]) == 295 + missing = next(r for r in result["rows"] if r["stack_kind"] == "missing") + assert missing["category"] == "dynamic_live_at_event" and missing["size_bytes"] == 125 + + +@pytest.mark.parametrize("template", ["active_blocks_at_event", "active_memory_callstack_at_event"]) +@pytest.mark.parametrize("event,slice_index", [(-1, None), (16, None), (8, 0), (8, True), (8, 99)]) +def test_point_event_rejects_boundary_outside_or_wrong_scope( + tmp_path, template, event, slice_index +): + root = case(tmp_path) + with SnapshotAnalyzer(root) as analyzer, pytest.raises(InvalidParameterError): + analyzer.execute_query(template, {"event_id": event}, slice_index=slice_index) + + +def test_active_block_limit_offset_exact_and_zero_top_n_keep_coverage(tmp_path): + root = case(tmp_path) + with SnapshotAnalyzer(root) as analyzer: + page = analyzer.execute_query( + "active_blocks_at_event", + {"event_id": 8, "order_by": "id", "order_dir": "ASC", "limit": 2, "offset": 1}, + max_rows=1, + exact_total=True, + ) + assert page["total"] == 9 and page["returned"] == 1 and page["truncated"] + assert page["rows"][0]["id"] == -20 + assert page["scope"]["source_coverage"]["active_blocks"] == 9 + zero = analyzer.execute_query( + "active_memory_callstack_at_event", {"event_id": 8, "top_n": 0}, exact_total=True + ) + assert {r["category"] for r in zero["rows"]} == {"static", "preexisting_live_at_event"} + assert sum(r["percent_of_active_blocks"] for r in zero["rows"]) == 100 + assert zero["has_more"] and zero["total"] == 7 + + +def add_frames(root, version=1, corrupt=None): + manifest = root / "manifest.json" + raw = json.loads(manifest.read_text()) + raw["extensions"] = {"ptSnapOrderedFrames": {"version": version}} + manifest.write_text(json.dumps(raw)) + frame = {"filename": "raw.py", "name": "same", "line": 1, "extra": {"native": True}} + for device in (0, 2): + for index in range(8): + with ( + closing(sqlite3.connect(root / f"device_{device}/slice_{index:05d}.db")) as conn, + conn, + ): + conn.executescript( + "CREATE TABLE pt_snap_frame_coverage (eventId INTEGER PRIMARY KEY,frameCount INTEGER); CREATE TABLE pt_snap_frame (eventId INTEGER,frameIndex INTEGER,frameJson TEXT, PRIMARY KEY(eventId,frameIndex));" + ) + for event in range(index * 2, index * 2 + 2): + conn.execute("INSERT INTO pt_snap_frame_coverage VALUES (?,3)", (event,)) + # order and repeats are significant. Events 0 and 1 share text + # but raw identity differs in the final frame. + frames = [frame, frame, {"name": f"event:{event}"}] + conn.executemany( + "INSERT INTO pt_snap_frame VALUES (?,?,?)", + [(event, i, json.dumps(f)) for i, f in enumerate(frames)], + ) + if index == 0: + if corrupt == "gap": + conn.execute("DELETE FROM pt_snap_frame WHERE eventId=0 AND frameIndex=1") + elif corrupt == "json": + conn.execute("UPDATE pt_snap_frame SET frameJson='[]' WHERE eventId=0") + elif corrupt == "absent": + conn.execute("DROP TABLE pt_snap_frame") + return frame + + +def test_recognized_ordered_frames_retain_order_repeats_exact_identity_and_shared_sources(tmp_path): + root = case(tmp_path) + frame = add_frames(root) + before = hashes(root) + with SnapshotAnalyzer(root) as analyzer: + row = analyzer.execute_query("event", {"id": 0, "stack_bytes": 0})["rows"][0] + assert row["frames"] == [frame, frame, {"name": "event:0"}] + assert row["frames_status"] == "ordered" and row["stack_truncated"] + result = analyzer.execute_query( + "active_memory_callstack_at_event", {"event_id": 8, "top_n": -1, "stack_bytes": 0} + ) + shared = [r for r in result["rows"] if r["stack_event_id"] in (0, 1)] + assert len(shared) == 2 and shared[0]["stack_id"] != shared[1]["stack_id"] + assert shared[0]["stack_id"] == row["stack_id"] + assert shared[0]["frames"][:2] == [frame, frame] + assert result["scope"]["source_coverage"]["ordered_frames_complete"] + assert ( + analyzer.get_database_overview()["dataset"]["structured_frames"] is False + ) # P0 validation, not scoped source coverage. + assert before == hashes(root) + + +@pytest.mark.parametrize( + "version,corrupt", [(999, None), (True, None), (1, "gap"), (1, "json"), (1, "absent")] +) +def test_unsupported_or_uncovered_frames_degrade_without_reconstructing_text( + tmp_path, version, corrupt +): + root = case(tmp_path) + add_frames(root, version, corrupt) + with SnapshotAnalyzer(root) as analyzer: + row = analyzer.execute_query("event", {"id": 0, "stack_bytes": 1000})["rows"][0] + assert row["frames"] is None and row["frames_status"] == "text_only" + assert row["callstack"].startswith("shared:") + result = analyzer.execute_query( + "active_memory_callstack_at_event", {"event_id": 8, "top_n": -1} + ) + assert not result["scope"]["source_coverage"]["ordered_frames_complete"] + + +def test_native_sources_preserve_sparse_ids_local_stack_collision_and_borrowed_lifetime(tmp_path): + request = dataclasses.replace(source_options(tmp_path, multiple=True), set_focus=False) + root = ImportService().import_snapshot(request).db_path + cache = ContextCache(maxsize=1) + try: + with SnapshotAnalyzer(root, context_cache=cache) as analyzer: + result = analyzer.execute_query( + "active_memory_callstack_at_event", {"event_id": 57, "top_n": -1} + ) + assert result["scope"]["first_event_id"] == 57 + assert result["scope"]["source_coverage"]["allocation_source_complete"] + blocks = analyzer.execute_query("active_blocks_at_event", {"event_id": 57}) + sources = [b["allocation_source"] for b in blocks["rows"] if b["allocation_source"]] + assert sources and all( + type(s["local_stack_id"]) is int and type(s["slice_index"]) is int for s in sources + ) + assert len(cache) == 1 + with pytest.raises(InvalidParameterError, match="sparse"): + analyzer.execute_query("active_blocks_at_event", {"event_id": 48}) + assert analyzer.execute_query("event", {"id": 48})["rows"] == [] + with pytest.raises(InvalidParameterError): + analyzer.execute_query("active_blocks_at_event", {"event_id": 58}) + held = next(iter(cache._entries.values()))[0] + assert len(cache) == 1 + with held.connect() as conn: + assert conn.execute("SELECT 1").fetchone()[0] == 1 + with pytest.raises(RuntimeError, match="closed"): + analyzer.execute_query("active_blocks_at_event", {"event_id": 57}) + finally: + cache.close() + + +@pytest.mark.parametrize("event", [7, 17, 27, 37, 47, 57, 67, 77, 87]) +def test_native_same_event_active_sets_bytes_and_attribution_match_standalone(tmp_path, event): + request = dataclasses.replace(source_options(tmp_path), set_focus=False) + service = ImportService() + root = service.import_snapshot(request).db_path + single = service.import_snapshot(dataclasses.replace(request, events_per_slice=None)).db_path + with SnapshotAnalyzer(root) as dataset, SnapshotAnalyzer(single) as reference: + params = {"event_id": event} + blocks = dataset.execute_query("active_blocks_at_event", params)["rows"] + original = reference.execute_query("active_blocks_at_event", params)["rows"] + # Unknown negative lifetimes are stable WITHIN a dataset, not globally + # comparable to a separately imported standalone DB's negative tokens. + from collections import Counter + + assert { + (r["id"], r["size"], r["allocEventId"], r["freeEventId"]) + for r in blocks + if r["allocEventId"] >= 0 + } == { + (r["id"], r["size"], r["allocEventId"], r["freeEventId"]) + for r in original + if r["allocEventId"] >= 0 + } + assert Counter( + (r["address"], r["size"], r["requestedSize"], r["allocEventId"], r["freeEventId"]) + for r in blocks + if r["allocEventId"] == -1 + ) == Counter( + (r["address"], r["size"], r["requestedSize"], r["allocEventId"], r["freeEventId"]) + for r in original + if r["allocEventId"] == -1 + ) + assert ( + sum(r["size"] for r in blocks) + == dataset.execute_query("event", {"id": event})["rows"][0]["active"] + ) + params["top_n"] = -1 + assert normalized_groups( + dataset.execute_query("active_memory_callstack_at_event", params)["rows"] + ) == normalized_groups( + reference.execute_query("active_memory_callstack_at_event", params)["rows"] + ) + + +@pytest.mark.parametrize("ordered", [False, True]) +def test_cross_owner_same_full_stack_merges_before_top_n_like_standalone(tmp_path, ordered): + from pt_snap_cli.core.models import ImportOptions + from tests.snapshot.test_sharded_replay import block, event, segment, write_source + + data = { + "segments": [ + segment(0, 64, [block(1000, 16, 16), block(1016, 16, 16), block(1032, 24, 24)]) + ], + "device_traces": [ + [ + event("alloc", addr=1000), + event("alloc", addr=1016), + event("alloc", addr=1032, size=24), + ] + ], + } + data["device_traces"][0][2]["frames"] = [{"filename": "other.py", "line": 8, "name": "other"}] + source = write_source(tmp_path, data) + service = ImportService() + request = ImportOptions(source, tmp_path / "outputs", set_focus=False, events_per_slice=1) + root = service.import_snapshot(request).db_path + single = service.import_snapshot(dataclasses.replace(request, events_per_slice=None)).db_path + if ordered: + raw = json.loads((root / "manifest.json").read_text()) + raw["extensions"] = {"ptSnapOrderedFrames": {"version": 1}} + import hashlib + + for i, item in enumerate(raw["devices"]["0"]["slices"]): + path = root / item["file"] + with closing(sqlite3.connect(path)) as conn, conn: + conn.executescript( + "CREATE TABLE pt_snap_frame_coverage (eventId INTEGER PRIMARY KEY,frameCount INTEGER); CREATE TABLE pt_snap_frame (eventId INTEGER,frameIndex INTEGER,frameJson TEXT, PRIMARY KEY(eventId,frameIndex));" + ) + conn.execute("INSERT INTO pt_snap_frame_coverage VALUES (?,1)", (i,)) + conn.execute( + "INSERT INTO pt_snap_frame VALUES (?,0,?)", + (i, json.dumps(data["device_traces"][0][i]["frames"][0])), + ) + item["sha256"] = hashlib.sha256(path.read_bytes()).hexdigest() + (root / "manifest.json").write_text(json.dumps(raw)) + before = hashes(root) + with SnapshotAnalyzer(root) as dataset, SnapshotAnalyzer(single) as reference: + params = {"event_id": 2, "top_n": 1} + result = dataset.execute_query("active_memory_callstack_at_event", params) + expected = reference.execute_query("active_memory_callstack_at_event", params) + assert normalized_groups(result["rows"]) == normalized_groups(expected["rows"]) + assert result["rows"][0]["block_count"] == 2 and result["rows"][0]["size_bytes"] == 32 + assert result["rows"][0]["percent_of_active_blocks"] == 100 + assert before == hashes(root) + + +@pytest.mark.parametrize("missing", [None, ""]) +def test_ordered_identity_ignores_formatted_text_availability_across_owners(tmp_path, missing): + root = case(tmp_path) + add_frames(root) + with closing(sqlite3.connect(root / "device_0/slice_00000.db")) as conn, conn: + conn.execute("UPDATE trace_entry_0 SET callstack=? WHERE id=0", (missing,)) + with closing(sqlite3.connect(root / "device_0/slice_00001.db")) as conn, conn: + conn.execute( + "UPDATE pt_snap_frame SET frameJson=? WHERE eventId=3 AND frameIndex=2", + (json.dumps({"name": "event:0"}),), + ) + before = hashes(root) + with SnapshotAnalyzer(root) as analyzer: + blocks = analyzer.execute_query("active_blocks_at_event", {"event_id": 8})["rows"] + first = next(b["allocation_source"] for b in blocks if b["id"] == 0) + other = next(b["allocation_source"] for b in blocks if b["id"] == 3) + assert first["frames"] == other["frames"] + assert first["stack_id"] == other["stack_id"] + result = analyzer.execute_query( + "active_memory_callstack_at_event", {"event_id": 8, "top_n": 1, "stack_bytes": 0} + ) + group = result["rows"][0] + assert (group["size_bytes"], group["block_count"]) == (130, 2) + assert group["stack_event_id"] == 3 + assert group["text_kind"] == "captured" and first["text_kind"] == "missing" + assert first["event"]["callstack"] == missing + row = analyzer.execute_query("event", {"id": 0, "stack_bytes": 0})["rows"][0] + assert row["frames"] == first["frames"] and row["text_kind"] == "missing" + json.dumps(row, allow_nan=False) + assert before == hashes(root) + + +@pytest.mark.parametrize( + "raw", + ['{"line":1e309}', '{"nested":' + "[" * 1200 + "0" + "]" * 1200 + "}"], + ids=["nonfinite", "too_deep"], +) +def test_nonfinite_and_deep_optional_frames_degrade_readonly(tmp_path, raw): + root = case(tmp_path) + add_frames(root) + with closing(sqlite3.connect(root / "device_0/slice_00000.db")) as conn, conn: + conn.execute( + "UPDATE pt_snap_frame SET frameJson=? WHERE eventId=0 AND frameIndex=0", (raw,) + ) + before = hashes(root) + with SnapshotAnalyzer(root) as analyzer: + row = analyzer.execute_query("event", {"id": 0})["rows"][0] + assert row["frames"] is None and row["frames_status"] == "text_only" + json.dumps(row, allow_nan=False) + assert before == hashes(root) + + +def test_native_bare_local_stack_ids_do_not_alias_foreign_sources(tmp_path): + root = ( + ImportService() + .import_snapshot(dataclasses.replace(source_options(tmp_path), set_focus=False)) + .db_path + ) + raw = json.loads((root / "manifest.json").read_text()) + for event in (17, 67): + item = next( + s + for s in raw["devices"]["0"]["slices"] + if s["startEventId"] <= event <= s["endEventId"] + ) + path = root / item["file"] + with closing(sqlite3.connect(path)) as conn, conn: + local = conn.execute( + "SELECT callstackId FROM trace_entry_0 WHERE id=?", (event,) + ).fetchone()[0] + conn.execute("UPDATE callstack SET id=42 WHERE id=?", (local,)) + conn.execute("UPDATE trace_entry_0 SET callstackId=42 WHERE callstackId=?", (local,)) + import hashlib + + item["sha256"] = hashlib.sha256(path.read_bytes()).hexdigest() + # Different complete texts with the same local ID must NOT alias. + for item in raw["devices"]["0"]["slices"]: + path = root / item["file"] + with closing(sqlite3.connect(path)) as conn, conn: + conn.execute( + "UPDATE callstack SET callstack='different complete stack' WHERE id IN (SELECT callstackId FROM trace_entry_0 WHERE id=67)" + ) + conn.execute( + "UPDATE pt_snap_block_reference SET allocCallstack='different complete stack' WHERE blockId=67" + ) + item["sha256"] = hashlib.sha256(path.read_bytes()).hexdigest() + (root / "manifest.json").write_text(json.dumps(raw)) + cache = ContextCache(maxsize=1) + try: + resolver = DatasetSourceResolver( + DatasetResolver().inspect(root), 0, cache, QueryBudget(None, time.monotonic()) + ) + sources = resolver.events([17, 67]) + assert len(sources) == 2 + assert all(s.local_stack_id == 42 for s in sources.values()) + assert sources[17].slice_index != sources[67].slice_index + assert sources[17].event["callstack"] != sources[67].event["callstack"] + assert sources[17].stack_id != sources[67].stack_id + assert {s.event["id"] for s in sources.values()} == {17, 67} + finally: + cache.close() + + +def test_high_block_count_batches_distinct_sources_and_bounds_cache_queries(tmp_path): + root = make_dataset(tmp_path / "large", devices=(0,), slices=300) + with closing(sqlite3.connect(root / "device_0/slice_00299.db")) as conn, conn: + conn.executemany( + "INSERT INTO block_0 VALUES (?,100,8,8,1,?,-1)", + [(event, event) for event in range(599)], + ) + dataset = DatasetResolver().inspect(root) + cache = ContextCache(maxsize=2) + try: + sources = DatasetSourceResolver(dataset, 0, cache, QueryBudget(None, time.monotonic())) + result = sources.events([*range(599), *range(599)]) + assert len(result) == 599 + assert sources.query_count == 300 # Per owning shard, not each block. + assert len(cache) <= 2 + with SnapshotAnalyzer(root, context_cache=cache) as analyzer: + high = analyzer.execute_query( + "active_memory_callstack_at_event", {"event_id": 599, "top_n": -1} + ) + assert sum(r["block_count"] for r in high["rows"]) == 599 + assert high["scope"]["source_coverage"]["source_queries"] == 302 + assert len(cache) <= 2 + # A single large shard specifically crosses the parameter batch bound. + single = make_dataset(tmp_path / "wide", devices=(0,), slices=1) + path = single / "device_0/slice_00000.db" + count = SOURCE_BATCH_SIZE * 3 + 7 + with closing(sqlite3.connect(path)) as conn, conn: + conn.executemany( + "INSERT INTO trace_entry_0 VALUES (?,4,100,8,0,8,8,16,'wide')", + [(i,) for i in range(2, count)], + ) + raw = json.loads((single / "manifest.json").read_text()) + raw["eventsPerSlice"] = count + raw["devices"]["0"]["eventCount"] = count + raw["devices"]["0"]["slices"][0]["endEventId"] = count - 1 + (single / "manifest.json").write_text(json.dumps(raw)) + resolver = DatasetSourceResolver( + DatasetResolver().inspect(single), 0, cache, QueryBudget(None, time.monotonic()) + ) + assert len(resolver.events([*range(count), *range(count)])) == count + assert resolver.query_count == 4 + finally: + cache.close() + + +def test_global_budget_counts_source_work_and_clears_sqlite_progress_handler(tmp_path, monkeypatch): + root = case(tmp_path) + cache = ContextCache(maxsize=1) + service = QueryService(context_cache=cache) + try: + original = DatasetSourceResolver.read + values = [] + + def counted(self, path, sql, params=None): + values.append(self.budget.remaining()) + return original(self, path, sql, params) + + monkeypatch.setattr(DatasetSourceResolver, "read", counted) + service.execute_query( + "active_memory_callstack_at_event", + {"event_id": 8}, + root, + timeout_s=10, + exact_total=True, + ) + assert len(values) > 3 and values == sorted(values, reverse=True) + expired = DatasetSourceResolver( + DatasetResolver().inspect(root), 0, cache, QueryBudget(0.001, time.monotonic() - 1) + ) + with pytest.raises(QueryTimeoutError): + expired.events([0, 1]) + sources = DatasetSourceResolver( + DatasetResolver().inspect(root), 0, cache, QueryBudget(0.0001, time.monotonic()) + ) + path = root / "device_0/slice_00000.db" + with pytest.raises(QueryTimeoutError): + sources.read( + path, + "WITH RECURSIVE n(x) AS (SELECT 1 UNION ALL SELECT x+1 FROM n WHERE x<10000000) SELECT SUM(x) FROM n", + ) + sources = DatasetSourceResolver( + DatasetResolver().inspect(root), 0, cache, QueryBudget(None, time.monotonic()) + ) + assert sources.read(path, "SELECT 1 AS okay") == [{"okay": 1}] + service.close() + assert len(cache) == 1 # Borrowed ownership, even following cancellation. + finally: + service.close() + cache.close() + + +def test_report_uses_same_point_event_core_and_global_aggregates_stay_explicit(tmp_path): + root = case(tmp_path) + report = ReportService() + try: + attribution = report.event_attribution(8, root, limit=-1) + with SnapshotAnalyzer(root) as analyzer: + assert dataclasses.asdict(attribution) == analyzer.execute_query( + "active_memory_callstack_at_event", + { + "event_id": 8, + "include_static": True, + "min_size": 0, + "top_n": -1, + "stack_bytes": -1, + }, + ) + for template in ( + "memory_peak", + "allocator_gap", + "leak_detection", + "block", + "callstack_analysis", + ): + assert analyzer.execute_query(template)["scope"]["range_complete"] + peak = report.peak_memory_report(root) + assert peak.event_id is not None and peak.source_coverage["allocation_source_complete"] + finally: + report.close() diff --git a/tests/core/test_dataset_global.py b/tests/core/test_dataset_global.py new file mode 100644 index 0000000..82a0dc5 --- /dev/null +++ b/tests/core/test_dataset_global.py @@ -0,0 +1,549 @@ +"""Device-global real SQLite regressions; synthetic inputs, never an upstream producer.""" + +import copy +import dataclasses +import hashlib +import json +import sqlite3 +import time +from contextlib import closing +from pathlib import Path + +import pytest +from typer.testing import CliRunner + +from pt_snap_cli.api import SnapshotAnalyzer +from pt_snap_cli.cli import app +from pt_snap_cli.core.context_cache import ContextCache +from pt_snap_cli.core.dataset_resolver import DatasetResolver +from pt_snap_cli.core.dataset_sources import DatasetSourceResolver, QueryBudget +from pt_snap_cli.core.dataset_support import DATASET_SUPPORT +from pt_snap_cli.core.errors import ( + InvalidParameterError, + QueryExecutionError, + QueryTimeoutError, + TemplateNotFoundError, +) +from pt_snap_cli.core.import_service import ImportService +from pt_snap_cli.core.query_service import QueryService +from pt_snap_cli.core.report_service import ReportService +from pt_snap_cli.query.registry import get_query, register_query +from tests.core.test_dataset_attribution import add_frames, case, standalone +from tests.core.test_dataset_import import source_options +from tests.test_dataset_focus import hashes, make_dataset + + +@pytest.fixture(autouse=True) +def isolate(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + monkeypatch.setenv("HOME", str(tmp_path)) + monkeypatch.setattr(Path, "home", lambda: tmp_path) + monkeypatch.delenv("PT_SNAP_DB_PATH", raising=False) + monkeypatch.delenv("PT_SNAP_QUERY_TIMEOUT", raising=False) + + +def projected(rows, keys): + return [{k: row[k] for k in keys} for row in rows] + + +@pytest.mark.parametrize( + "template", + [ + "memory_peak", + "allocator_gap", + "allocation", + "event", + "block", + "leak_detection", + "freed_block_lifetime", + "callstack_analysis", + "preexisting_live", + ], +) +def test_inline_global_matches_real_standalone_without_extensions(tmp_path, template): + root = case(tmp_path) + single = standalone(root, tmp_path) + before = hashes(root) + params = ( + {"min_count": 1} + if template == "callstack_analysis" + else {"event_id": 8} if template == "preexisting_live" else {} + ) + with SnapshotAnalyzer(root) as dataset, SnapshotAnalyzer(single) as reference: + actual = dataset.execute_query(template, params, exact_total=True) + expected = reference.execute_query(template, params, exact_total=True) + keys = list(expected["rows"][0]) if expected["rows"] else [] + assert projected(actual["rows"], keys) == expected["rows"] + assert actual["total"] == expected["total"] and actual["total_is_exact"] + assert actual["scope"]["range_complete"] + assert actual["scope"]["boundary_events_included"] is False + assert hashes(root) == before + + +@pytest.mark.parametrize( + "template", + [ + "memory_peak", + "allocator_gap", + "allocation", + "event", + "block", + "leak_detection", + "freed_block_lifetime", + "callstack_analysis", + ], +) +def test_native_sparse_multiple_devices_matches_standalone(tmp_path, template): + request = dataclasses.replace(source_options(tmp_path, multiple=True), set_focus=False) + service = ImportService() + root = service.import_snapshot(request).db_path + single = service.import_snapshot(dataclasses.replace(request, events_per_slice=None)).db_path + before = hashes(root) + params = ( + {"min_count": 1} + if template == "callstack_analysis" + else {"min_id": 7} if template in ("event", "allocation") else {} + ) + with SnapshotAnalyzer(root) as dataset, SnapshotAnalyzer(single) as reference: + for device in (0, 1): + actual = dataset.execute_query(template, params, device_id=device, exact_total=True) + expected = reference.execute_query(template, params, device_id=device, exact_total=True) + keys = list(expected["rows"][0]) if expected["rows"] else [] + # Negative IDs are dataset-local tokens, not comparable across imports. + if template == "block": + keys.remove("id") + # Stored state is an observation, not a lifecycle invariant. + # The standalone dump and latest containing slice can observe + # different pending-free states; neither is state at arbitrary E. + keys.remove("state") + assert all(r["state_scope"] == "latest_slice_observation" for r in actual["rows"]) + + def norm(rs, fields=tuple(keys)): + return sorted(tuple(str(r[k]) for k in fields) for r in rs) + + assert norm(actual["rows"]) == norm(expected["rows"]) + else: + assert projected(actual["rows"], keys) == expected["rows"] + assert actual["total"] == expected["total"] + assert hashes(root) == before + + +def test_peaks_ties_last_shard_boundary_and_same_event_report(tmp_path): + root = case(tmp_path) + single = standalone(root, tmp_path) + for index in range(8): + with closing(sqlite3.connect(root / f"device_0/slice_{index:05d}.db")) as conn, conn: + conn.execute( + "UPDATE trace_entry_0 SET allocated=99999,active=99999,reserved=99999 WHERE id<0" + ) + conn.execute("UPDATE trace_entry_0 SET reserved=5000 WHERE id IN (4,15)") + conn.execute("UPDATE trace_entry_0 SET allocated=3000 WHERE id=15") + with closing(sqlite3.connect(single)) as conn, conn: + conn.execute("UPDATE trace_entry_0 SET reserved=5000 WHERE id IN (4,15)") + conn.execute("UPDATE trace_entry_0 SET allocated=3000 WHERE id=15") + conn.execute("INSERT INTO trace_entry_0 VALUES (-1,2,0,0,0,99999,99999,99999,'fake')") + report = ReportService() + try: + with SnapshotAnalyzer(root) as dataset, SnapshotAnalyzer(single) as reference: + peak = dataset.execute_query("memory_peak")["rows"][0] + assert peak == reference.execute_query("memory_peak")["rows"][0] + assert peak["peak_allocated_event_id"] == 15 and peak["peak_reserved_event_id"] == 4 + assert ( + dataset.execute_query("allocator_gap")["rows"] + == reference.execute_query("allocator_gap")["rows"] + ) + for metric in ("allocated", "active", "reserved"): + result = report.peak_memory_report(root, metric=metric, limit=-1, timeout_s=10) + original = report.peak_memory_report(single, metric=metric, limit=-1, timeout_s=10) + assert result.event_id == original.event_id + assert result.active_bytes_at_event == original.active_bytes_at_event + assert result.included_bytes == original.included_bytes + assert result.coverage_percent == original.coverage_percent + assert result.allocator_gap == original.allocator_gap + assert result.source_coverage["allocation_source_complete"] + assert result.budget_scope == "report_composition" + finally: + report.close() + + +def test_global_stack_winner_threshold_weighted_average_and_all_actions(tmp_path): + root = make_dataset(tmp_path / "ranking", devices=(0,), slices=3) + for index in range(3): + with closing(sqlite3.connect(root / f"device_0/slice_{index:05d}.db")) as conn, conn: + conn.execute( + "UPDATE trace_entry_0 SET callstack=?,size=11 WHERE id=?", + (f"local:{index}", index * 2), + ) + conn.execute( + "UPDATE trace_entry_0 SET callstack='global',size=?,action=6 WHERE id=?", + (10 + index, index * 2 + 1), + ) + with SnapshotAnalyzer(root) as analyzer: + result = analyzer.execute_query( + "callstack_analysis", {"min_count": 2, "min_size": 30, "limit": 1}, exact_total=True + ) + assert result["total"] == 1 and not result["truncated"] + row = result["rows"][0] + assert ( + row["callstack"], + row["alloc_count"], + row["total_size"], + row["avg_size"], + row["max_size"], + ) == ("global", 3, 33, 11, 12) + assert result["semantics_version"] == 2 # counts free_completed too, not allocations + assert ( + analyzer.execute_query("callstack_analysis", {"min_count": 1, "limit": 1})["rows"][0][ + "callstack" + ] + == "global" + ) + + +def test_lifecycles_terminal_survival_completed_free_reuse_and_unknown(tmp_path): + root = case(tmp_path) + # An allocation present in early shards but absent at the terminal shard is + # NOT a terminal candidate, even if its earlier observation says unfreed. + with closing(sqlite3.connect(root / "device_0/slice_00007.db")) as conn, conn: + conn.execute("DELETE FROM block_0 WHERE id=3") + with SnapshotAnalyzer(root) as analyzer: + all_rows = analyzer.execute_query("block", exact_total=True) + assert all_rows["total"] == 10 # not 8 * 10 + assert len({r["lifecycle_id"] for r in all_rows["rows"]}) == 10 + same_address = [r for r in all_rows["rows"] if r["address"] == 100] + assert ( + len(same_address) == 2 + and same_address[0]["lifecycle_id"] != same_address[1]["lifecycle_id"] + ) + candidates = analyzer.execute_query("leak_detection")["rows"] + assert {r["id"] for r in candidates} == {4, 5, 6, 7, 11} + assert all(r["terminal_survivor"] for r in candidates) + freed = analyzer.execute_query("freed_block_lifetime")["rows"] + assert sum(r["block_count"] for r in freed) == 2 + old = next(r for r in all_rows["rows"] if r["id"] == 0) + assert ( + old["free_source"]["event"]["id"] == 10 and old["free_source"]["event"]["action"] == 6 + ) + assert all( + r["identity_status"] == "stable_negative_token" for r in all_rows["rows"] if r["id"] < 0 + ) + with closing(sqlite3.connect(root / "device_0/slice_00000.db")) as conn, conn: + conn.execute("UPDATE trace_entry_0 SET action=2 WHERE id=0") + with SnapshotAnalyzer(root) as analyzer: + result = analyzer.execute_query("block") + assert result["scope"]["source_coverage"]["unproved_identities"] == 8 + assert not result["scope"]["source_coverage"]["allocation_source_complete"] + assert all(r["id"] != 0 for r in analyzer.execute_query("leak_detection")["rows"]) + + +def test_global_page_filters_ties_exact_total_source_range_and_empty(tmp_path): + root = case(tmp_path) + single = standalone(root, tmp_path) + params = { + "min_id": 1, + "max_id": 14, + "min_active": 50, + "order_by": "active", + "order_dir": "DESC", + "offset": 2, + "limit": 3, + } + with SnapshotAnalyzer(root) as dataset, SnapshotAnalyzer(single) as reference: + actual = dataset.execute_query("event", params, max_rows=2, exact_total=True) + expected = reference.execute_query("event", params, max_rows=2, exact_total=True) + assert actual["rows"] == expected["rows"] and actual["total"] == expected["total"] + assert len(actual["rows"][0]) == 9 + assert actual["has_more"] and actual["truncated"] + assert actual["scope"]["slice_indices"] == list(range(8)) + assert actual["scope"]["source_coverage"]["events_resolved"] == 2 + assert actual["scope"]["range_complete"] + empty = dataset.execute_query("event", {"action": 99}, exact_total=True) + assert empty["total"] == 0 and not empty["truncated"] + for endpoints in ({"min_id": -1}, {"max_id": 99}, {"min_id": 3, "max_id": 2}, {"id": True}): + with pytest.raises(InvalidParameterError): + dataset.execute_query("event", endpoints) + request = dataclasses.replace(source_options(tmp_path), set_focus=False) + native = ImportService().import_snapshot(request).db_path + with SnapshotAnalyzer(native) as analyzer: + gap = analyzer.execute_query("memory_peak", {"start_id": 18, "end_id": 26}) + assert all(v is None for v in gap["rows"][0].values()) + assert gap["scope"]["actual_range"]["real_event_count"] == 0 + assert analyzer.execute_query("event", {"min_id": 18, "max_id": 26})["rows"] == [] + + +def test_runtime_overrides_custom_sql_and_capabilities_are_not_builtin_semantics(tmp_path): + root = case(tmp_path) + original = get_query("memory_peak") + custom = copy.deepcopy(original) + custom.query = "SELECT 999 AS fake" + register_query(custom) + try: + with SnapshotAnalyzer(root) as analyzer: + assert ( + analyzer.get_template_info("memory_peak")["dataset_support"]["supported"] is False + ) + with pytest.raises(QueryExecutionError, match="runtime override"): + analyzer.execute_query("memory_peak") + with pytest.raises(TemplateNotFoundError): + analyzer.execute_query("SELECT * FROM block_0") + finally: + register_query(original) + service = QueryService() + try: + member = root / "device_0/slice_00000.db" + executor = service._get_executor(service._validated_context(member)) + executor.register_template(custom) + assert service.get_template_info("memory_peak").dataset_support["supported"] is False + with pytest.raises(QueryExecutionError, match="runtime override"): + service.execute_query("memory_peak", db_path=root) + finally: + service.close() + with SnapshotAnalyzer(root) as analyzer: + contracts = analyzer.list_capabilities()["templates"] + assert {c["name"] for c in contracts if c["dataset_support"]["supported"]} == set( + DATASET_SUPPORT + ) + + +def test_work_limits_detached_batches_borrowed_lru_timeout_cleanup_and_report_deadline( + tmp_path, monkeypatch +): + root = case(tmp_path) + cache = ContextCache(maxsize=1) + service = QueryService(context_cache=cache) + report = ReportService() + try: + budget = QueryBudget(None, time.monotonic(), max_work_rows=4) + with pytest.raises(QueryExecutionError, match="work budget"): + service.execute_query("block", db_path=root, _budget=budget) + # The entire real range consumes 16 rows before final max_rows=1; + # its source lookup must use the SAME cumulative budget, not reset. + budget = QueryBudget(None, time.monotonic(), max_work_rows=16) + with pytest.raises(QueryExecutionError, match="work budget"): + service.execute_query( + "event", db_path=root, max_rows=1, exact_total=True, _budget=budget + ) + budget = QueryBudget(None, time.monotonic(), max_work_bytes=1) + with pytest.raises(QueryExecutionError, match="work budget"): + service.execute_query("event", db_path=root, _budget=budget) + assert service.execute_query("block", db_path=root).rows and len(cache) == 1 + held = next(iter(cache._entries.values()))[0] + with held.connect() as conn: + assert conn.execute("SELECT 1").fetchone()[0] == 1 + remaining, budgets = [], [] + original = DatasetSourceResolver.read + + def counted(self, path, sql, values=None): + remaining.append(self.budget.remaining()) + budgets.append(self.budget) + return original(self, path, sql, values) + + monkeypatch.setattr(DatasetSourceResolver, "read", counted) + report.peak_memory_report(root, timeout_s=10) + assert len({id(b) for b in budgets}) == 1 + assert remaining == sorted(remaining, reverse=True) + remaining.clear() + expired = QueryBudget(0.001, time.monotonic() - 1) + with pytest.raises(QueryTimeoutError): + service.execute_query("event", db_path=root, _budget=expired) + source = DatasetSourceResolver( + DatasetResolver().inspect(root), 0, cache, QueryBudget(0.001, time.monotonic()) + ) + with pytest.raises(QueryTimeoutError): + source.read( + root / "device_0/slice_00000.db", + "WITH RECURSIVE n(x) AS (SELECT 1 UNION ALL SELECT x+1 FROM n WHERE x<10000000) SELECT SUM(x) FROM n", + ) + source = DatasetSourceResolver( + DatasetResolver().inspect(root), 0, cache, QueryBudget(None, time.monotonic()) + ) + with pytest.raises(QueryExecutionError): + source.read(root / "device_0/slice_00000.db", "SELECT * FROM absent_table") + assert source.read(root / "device_0/slice_00000.db", "SELECT 1 AS okay") == [{"okay": 1}] + service.close() + assert len(cache) == 1 + finally: + service.close() + report.close() + cache.close() + + +def test_callback_deadline_keeps_domain_timeout_and_cli_json_code(tmp_path, monkeypatch): + root = case(tmp_path) + original = QueryBudget.consume + + def expire(self, rows): + self.started -= 100 + return original(self, rows) + + cache = ContextCache(maxsize=1) + try: + with monkeypatch.context() as patch: + patch.setattr(QueryBudget, "consume", expire) + with SnapshotAnalyzer(root, context_cache=cache) as analyzer: + with pytest.raises(QueryTimeoutError): + analyzer.execute_query("memory_peak", timeout_s=10) + cli = CliRunner().invoke( + app, + ["query", str(root), "--template-use", "memory_peak", "--timeout", "10", "--json"], + ) + assert cli.exit_code != 0 and not cli.stdout + assert json.loads(cli.stderr)["error"]["code"] == "QUERY_TIMEOUT" + with SnapshotAnalyzer(root, context_cache=cache) as analyzer: + assert analyzer.execute_query("memory_peak")["rows"] + assert len(cache) == 1 + finally: + cache.close() + + +def test_ordered_arrays_control_global_stack_identity_not_local_ids_or_display(tmp_path): + root = case(tmp_path) + add_frames(root) + with SnapshotAnalyzer(root) as analyzer: + rows = analyzer.execute_query("callstack_analysis", {"min_count": 1})["rows"] + assert len(rows) == 16 # every explicit raw array differs, including repeated text + assert len({r["source_stack_id"] for r in rows}) == 16 + assert all(r["frames_status"] == "ordered" for r in rows) + + +def test_raw_stats_identity_prefers_captured_text_over_null_and_empty(tmp_path): + root = case(tmp_path) + add_frames(root) + for index, changes in [(0, [(0, None), (1, "")]), (1, [(3, "third")])]: + with closing(sqlite3.connect(root / f"device_0/slice_{index:05d}.db")) as conn, conn: + for event, text in changes: + conn.execute("UPDATE trace_entry_0 SET callstack=? WHERE id=?", (text, event)) + conn.execute( + "UPDATE pt_snap_frame SET frameJson=? WHERE eventId=? AND frameIndex=2", + (json.dumps({"name": "event:0"}), event), + ) + with SnapshotAnalyzer(root) as analyzer: + identity = analyzer.execute_query("event", {"id": 0})["rows"][0]["source_stack_id"] + row = next( + r + for r in analyzer.execute_query("callstack_analysis", {"min_count": 1})["rows"] + if r["source_stack_id"] == identity + ) + assert row["alloc_count"] == 3 and row["callstack"] == "third" + assert row["stack_event_id"] == 3 and row["text_kind"] == "captured" + assert analyzer.execute_query("event", {"id": 1})["rows"][0]["text_kind"] == "missing" + + +@pytest.mark.parametrize("layout", ["inline", "native"]) +@pytest.mark.parametrize("detailed", [False, True]) +def test_all_dataset_row_keys_are_declared_by_shared_template_schema(tmp_path, layout, detailed): + if layout == "inline": + root = case(tmp_path) + event = 15 + if detailed: + add_frames(root) + else: + root = ( + ImportService() + .import_snapshot(dataclasses.replace(source_options(tmp_path), set_focus=False)) + .db_path + ) + event = 87 + if detailed: + manifest = root / "manifest.json" + raw = json.loads(manifest.read_text()) + raw["extensions"] = {"ptSnapOrderedFrames": {"version": 1}} + for device, data in raw["devices"].items(): + for item in data["slices"]: + path = root / item["file"] + with closing(sqlite3.connect(path)) as conn, conn: + conn.executescript( + "CREATE TABLE pt_snap_frame_coverage (eventId INTEGER PRIMARY KEY,frameCount INTEGER);" + "CREATE TABLE pt_snap_frame (eventId INTEGER,frameIndex INTEGER,frameJson TEXT, PRIMARY KEY(eventId,frameIndex));" + ) + ids = [ + r[0] + for r in conn.execute( + f"SELECT id FROM trace_entry_{device} WHERE id>=0" + ) + ] + conn.executemany( + "INSERT INTO pt_snap_frame_coverage VALUES (?,1)", [(i,) for i in ids] + ) + conn.executemany( + "INSERT INTO pt_snap_frame VALUES (?,0,?)", + [(i, '{"name":"same","line":1}') for i in ids], + ) + item["sha256"] = hashlib.sha256(path.read_bytes()).hexdigest() + manifest.write_text(json.dumps(raw)) + params = { + "event": {"stack_bytes": 0 if detailed else -1}, + "callstack_analysis": {"min_count": 1}, + "active_blocks_at_event": {"event_id": event}, + "active_memory_callstack_at_event": { + "event_id": event, + "top_n": -1, + "stack_bytes": 0 if detailed else -1, + }, + "preexisting_live": {"event_id": event}, + } + with SnapshotAnalyzer(root) as analyzer: + for template in DATASET_SUPPORT: + rows = analyzer.execute_query(template, params.get(template), max_rows=1)["rows"] + info = analyzer.get_template_info(template) + columns = {field["column"] for field in info["output_schema"]} + assert all(set(row) <= columns for row in rows), ( + template, + [set(row) - columns for row in rows], + ) + cli = CliRunner().invoke(app, ["query", "--template-info", template, "--json"]) + assert cli.exit_code == 0, cli.output + assert json.loads(cli.stdout)["output_schema"] == info["output_schema"] + if template in ("block", "leak_detection", "callstack_analysis"): + dataset_fields = [ + field + for field in info["output_schema"] + if any( + "Dataset only" in text for text in field.get("interpretation_limits", []) + ) + ] + assert dataset_fields and all( + field.get("metric_semantics") and field.get("scope") for field in dataset_fields + ) + + +def test_cli_api_report_global_parity_and_range_options(tmp_path): + root = case(tmp_path) + runner = CliRunner() + with SnapshotAnalyzer(root) as analyzer: + api = analyzer.execute_query("memory_peak", {"start_id": 4, "end_id": 15}) + cli = runner.invoke( + app, + [ + "query", + str(root), + "--template-use", + "memory_peak", + "--params", + '{"start_id":4,"end_id":15}', + "--json", + ], + ) + assert cli.exit_code == 0, cli.output + payload = json.loads(cli.stdout) + for key in api: + assert payload[key] == api[key] + cli = runner.invoke( + app, + [ + "report", + "peak-memory", + str(root), + "--start-id", + "4", + "--end-id", + "15", + "--timeout", + "10", + "--json", + ], + ) + assert cli.exit_code == 0, cli.output + result = json.loads(cli.stdout) + assert result["scope"]["requested_range"] == {"first_event_id": 4, "last_event_id": 15} + assert result["budget_scope"] == "report_composition" and result["timeout_s"] == 10 diff --git a/tests/core/test_dataset_import.py b/tests/core/test_dataset_import.py new file mode 100644 index 0000000..005c5a4 --- /dev/null +++ b/tests/core/test_dataset_import.py @@ -0,0 +1,429 @@ +"""Original deterministic temporary snapshots; no new committed pickle fixtures.""" + +import copy +import dataclasses +import hashlib +import json +import sqlite3 +from contextlib import closing +from pathlib import Path + +import pytest + +from pt_snap_cli.core import import_metadata, import_service, native_dataset_contract +from pt_snap_cli.core.dataset_import_backend import DatasetImportBackend +from pt_snap_cli.core.dataset_resolver import DatasetResolver +from pt_snap_cli.core.errors import ImportExecutionError, InvalidParameterError, SourceChangedError +from pt_snap_cli.core.focus_service import FocusService +from pt_snap_cli.core.import_service import ImportService +from pt_snap_cli.core.models import ImportOptions +from pt_snap_cli.core.native_dataset_contract import NATIVE_FORMAT +from pt_snap_cli.snapshot.tools.adaptors import sharded_replay +from tests.snapshot.test_sharded_replay import lifecycle_data, write_source + + +@pytest.fixture(autouse=True) +def isolate(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + monkeypatch.setenv("HOME", str(tmp_path)) + monkeypatch.setattr(Path, "home", lambda: tmp_path) + monkeypatch.delenv("PT_SNAP_DB_PATH", raising=False) + + +def source_options(tmp_path, *, multiple=False): + data = lifecycle_data() + for index, event in enumerate(data["device_traces"][0]): + event["id"] = index * 10 + 7 + if multiple: + data["device_traces"].append(copy.deepcopy(data["device_traces"][0])) + data["segments"] += [{**copy.deepcopy(s), "device": 1} for s in data["segments"]] + data["device_traces"].append([]) + return ImportOptions(write_source(tmp_path, data), tmp_path / "output", events_per_slice=2) + + +def hashes(root): + return { + str(p.relative_to(root)): hashlib.sha256(p.read_bytes()).hexdigest() + for p in root.rglob("*") + if p.is_file() + } + + +def saved_state(tmp_path): + service = ImportService() + options = source_options(tmp_path) + first = service.import_snapshot(options) + focus = tmp_path / ".pt-snap/focus.json" + return ( + service, + dataclasses.replace(options, force=True), + first, + hashes(first.db_path), + focus.read_bytes(), + ) + + +def assert_preserved(tmp_path, first, old, focus): + assert hashes(first.db_path) == old + assert (tmp_path / ".pt-snap/focus.json").read_bytes() == focus + assert list(first.db_path.parent.iterdir()) == [first.db_path] + + +def test_generation_complete_focus_result_and_all_devices(tmp_path): + options = source_options(tmp_path, multiple=True) + result = ImportService().import_snapshot(options) + assert result.dataset_path == result.db_path + assert result.devices == (0, 1) and result.omitted_devices == (2,) + assert result.slice_count == 10 and result.format == NATIVE_FORMAT + assert result.focus_state.callstack_layout == "v2" + dataset = DatasetResolver().inspect(result.db_path) + assert dataset.validation.real_event_count == 18 + assert dataset.validation.manifest.devices[0].event_count == 9 + assert ( + result.metadata.source_sha256 + == hashlib.sha256(options.snapshot_file.read_bytes()).hexdigest() + ) + manifest = json.loads((result.db_path / "manifest.json").read_text()) + assert "cacheHash" not in manifest and manifest["format"] == NATIVE_FORMAT + assert manifest["devices"]["0"]["slices"][-1]["endEventId"] == 87 + + +def test_cache_reuses_full_artifact_without_loading_or_cli_version_invalidation( + tmp_path, monkeypatch +): + options = dataclasses.replace(source_options(tmp_path), set_focus=False) + service = ImportService() + first = service.import_snapshot(options) + monkeypatch.setattr( + service._dataset_backend.replay, + "stage", + lambda *a, **k: pytest.fail("cache hit loaded source"), + ) + monkeypatch.setattr(import_metadata, "__version__", "future-cli-release") + second = service.import_snapshot(options) + assert second.reused and second.metadata == first.metadata + assert second.focus_state is None and not (tmp_path / ".pt-snap").exists() + + +@pytest.mark.parametrize( + "axis", + [ + "content", + "device", + "capacity", + "format", + "manifest", + "import", + "extension", + "metadata", + "db-layout", + ], +) +def test_cache_axes_require_force_and_rebuild(tmp_path, monkeypatch, axis): + service, options, first, old, focus = saved_state(tmp_path) + requested = dataclasses.replace(options, force=False) + if axis == "content": + options.snapshot_file.write_bytes(options.snapshot_file.read_bytes() + b"trailing-content") + elif axis == "device": + requested = dataclasses.replace(requested, device=0) + elif axis == "capacity": + requested = dataclasses.replace(requested, events_per_slice=3) + else: + constants = { + "format": (native_dataset_contract, "NATIVE_FORMAT_VERSION"), + "manifest": (native_dataset_contract, "NATIVE_MANIFEST_VERSION"), + "import": (native_dataset_contract, "NATIVE_IMPORT_VERSION"), + "extension": (native_dataset_contract, "NATIVE_EXTENSION_VERSION"), + "metadata": (import_metadata, "METADATA_SCHEMA_VERSION"), + "db-layout": (import_metadata, "IMPORT_FORMAT_VERSION"), + } + module, key = constants[axis] + # A manifest protocol upgrade cannot adopt/replace an unknown older schema. + # This axis is therefore exercised as identity invalidation, not schema relabeling. + if axis in ("manifest", "metadata", "db-layout"): + path = first.db_path / "manifest.json" + data = json.loads(path.read_text()) + identity_key = { + "manifest": "manifestContractVersion", + "metadata": "metadataSchemaVersion", + "db-layout": "importFormatVersion", + }[axis] + data["identity"][identity_key] += 1 + if axis != "manifest": + column = ( + "metadata_schema_version" if axis == "metadata" else "import_format_version" + ) + data["metadata"][column] += 1 + for device_data in data["devices"].values(): + for shard in device_data["slices"]: + member = first.db_path / shard["file"] + with closing(sqlite3.connect(member)) as conn, conn: + conn.execute( + f"UPDATE pt_snap_metadata SET {column}=?", + (data["metadata"][column],), + ) + shard["sha256"] = hashlib.sha256(member.read_bytes()).hexdigest() + path.write_text(json.dumps(data)) + old = hashes(first.db_path) + else: + monkeypatch.setattr(module, key, getattr(module, key) + 1) + with pytest.raises(ImportExecutionError, match="not reusable"): + service.import_snapshot(requested) + assert_preserved(tmp_path, first, old, focus) + result = service.import_snapshot(dataclasses.replace(requested, force=True)) + assert not result.reused and result.cache_miss_reason == "forced" + assert DatasetResolver().inspect(result.db_path) is not None + + +def test_corrupt_late_member_never_reuses_first_ready_shard(tmp_path): + service, options, first, _, focus = saved_state(tmp_path) + last = first.db_path / "device_0/slice_00004.db" + last.write_bytes(b"corrupt last member") + old = hashes(first.db_path) + with pytest.raises(ImportExecutionError, match="not reusable"): + service.import_snapshot(dataclasses.replace(options, force=False)) + assert_preserved(tmp_path, first, old, focus) + assert not service.import_snapshot(options).reused + + +@pytest.mark.parametrize( + "phase", + ["load", "write", "finalize", "registry-validation", "metadata", "manifest", "validation"], +) +def test_prepublication_failure_preserves_artifact_focus_and_closes_connections( + tmp_path, monkeypatch, phase +): + service, options, first, old, focus = saved_state(tmp_path) + handles = [] + connect = sqlite3.connect + + def counted(*args, **kwargs): + conn = connect(*args, **kwargs) + handles.append(conn) + return conn + + monkeypatch.setattr(sqlite3, "connect", counted) + + def fail(*args, **kwargs): + raise OSError(phase + " injected failure") + + if phase == "load": + monkeypatch.setattr( + "pt_snap_cli.core.sharded_replay_service.load_snapshot_representation", fail + ) + elif phase == "write": + monkeypatch.setattr(sharded_replay.SnapshotDbHandler, "insert_event", fail) + elif phase == "finalize": + monkeypatch.setattr(sharded_replay.BlockRegistry, "finalize", fail) + elif phase == "registry-validation": + monkeypatch.setattr(sharded_replay.BlockRegistry, "validate", fail) + elif phase == "metadata": + monkeypatch.setattr(service._metadata_service, "write", fail) + elif phase == "manifest": + monkeypatch.setattr(import_service.json, "dump", fail) + elif phase == "validation": + monkeypatch.setattr(import_service.DatasetResolver, "inspect", fail) + with pytest.raises(ImportExecutionError, match="injected failure"): + service.import_snapshot(options) + assert_preserved(tmp_path, first, old, focus) + for conn in handles: + with pytest.raises(sqlite3.ProgrammingError): + conn.execute("SELECT 1") + + +@pytest.mark.parametrize("move", ["old", "new"]) +@pytest.mark.parametrize("after_move", [False, True]) +def test_publication_rename_failure_even_after_mutation_compensates( + tmp_path, monkeypatch, move, after_move +): + service, options, first, old, focus = saved_state(tmp_path) + real = service._dataset_backend._move + fired = False + + def fault(source, dest, identity): + nonlocal fired + selected = ( + source == first.db_path + if move == "old" + else dest == first.db_path and source.name == "artifact" + ) + if selected and not fired: + fired = True + if after_move: + real(source, dest, identity) + raise OSError("publication injected failure") + real(source, dest, identity) + + monkeypatch.setattr(service._dataset_backend, "_move", fault) + with pytest.raises(ImportExecutionError, match="publication injected failure"): + service.import_snapshot(options) + assert_preserved(tmp_path, first, old, focus) + + +@pytest.mark.parametrize("existing", [False, True]) +def test_focus_mutates_before_raising_restores_exact_bytes_memory_and_artifact( + tmp_path, monkeypatch, existing +): + if existing: + service, options, first, old, focus = saved_state(tmp_path) + else: + service = ImportService() + options = source_options(tmp_path) + fs = FocusService() + fs._config._config = {"remember": ["old-memory-state"]} + service._focus_service = fs + + def mutate_then_fail(*a, **k): + path = tmp_path / ".pt-snap/focus.json" + path.write_bytes(b"mutated before raising") + fs._config._config["remember"].append("changed") + raise OSError("focus injected failure") + + monkeypatch.setattr(fs._config, "write_project_focus", mutate_then_fail) + with pytest.raises(ImportExecutionError, match="focus injected failure"): + service.import_snapshot(options) + assert fs._config._config == {"remember": ["old-memory-state"]} + if existing: + assert_preserved(tmp_path, first, old, focus) + else: + assert not (tmp_path / ".pt-snap/focus.json").exists() + assert not list(options.output_dir.iterdir()) + + +def test_source_change_after_validation_rejects_publication(tmp_path, monkeypatch): + service, options, first, old, focus = saved_state(tmp_path) + initial = service._metadata_service.calculate_sha256(options.snapshot_file) + sequence = iter([initial, "f" * 64]) + monkeypatch.setattr(service._metadata_service, "calculate_sha256", lambda p: next(sequence)) + with pytest.raises(SourceChangedError, match="source changed"): + service.import_snapshot(options) + assert_preserved(tmp_path, first, old, focus) + + +def test_directory_rollback_failure_keeps_recovery_bytes_and_reports(tmp_path, monkeypatch): + service, options, first, old, focus = saved_state(tmp_path) + real = service._dataset_backend._move + + def fault(source, dest, identity): + if source.name == "previous": + raise OSError("rollback injected failure") + real(source, dest, identity) + + monkeypatch.setattr(service._dataset_backend, "_move", fault) + monkeypatch.setattr( + service._focus_service._config, + "write_project_focus", + lambda *a, **k: (_ for _ in ()).throw(OSError("focus failed")), + ) + with pytest.raises(ImportExecutionError, match="rollback also failed.*Recovery directory"): + service.import_snapshot(options) + backup = list(options.output_dir.glob("*.recovery-*/previous")) + assert len(backup) == 1 and hashes(backup[0]) == old + assert (tmp_path / ".pt-snap/focus.json").read_bytes() == focus + + +def test_focus_rollback_failure_preserves_checkpoint_and_restores_old_dataset( + tmp_path, monkeypatch +): + service, options, first, old, focus = saved_state(tmp_path) + fs = service._focus_service + + def mutate(*a, **k): + (tmp_path / ".pt-snap/focus.json").write_bytes(b"changed") + raise OSError("focus failed") + + monkeypatch.setattr(fs._config, "write_project_focus", mutate) + monkeypatch.setattr( + fs, "_restore_project_focus", lambda *a: (_ for _ in ()).throw(OSError("restore failed")) + ) + with pytest.raises( + ImportExecutionError, match="focus rollback also failed.*Recovery focus checkpoint" + ): + service.import_snapshot(options) + backups = list((tmp_path / ".pt-snap").glob("*.recovery")) + assert len(backups) == 1 and backups[0].read_bytes() == focus + assert hashes(first.db_path) == old + + +@pytest.mark.parametrize("kind", ["empty", "unknown", "symlink", "dangling", "extra"]) +def test_force_never_deletes_unrecognized_targets_or_aliases(tmp_path, kind): + service = ImportService() + options = source_options(tmp_path) + target = DatasetImportBackend.target_path(options.snapshot_file, options.output_dir) + target.parent.mkdir() + if kind in ("symlink", "dangling"): + victim = tmp_path / "victim" + if kind == "symlink": + victim.mkdir() + (victim / "owner").write_text("safe") + target.symlink_to(victim, target_is_directory=True) + elif kind == "extra": + service.import_snapshot(options) + (target / "unrelated").write_text("preserve") + else: + target.mkdir() + if kind == "unknown": + (target / "owner").write_text("safe") + with pytest.raises(ImportExecutionError): + service.import_snapshot(dataclasses.replace(options, force=True)) + assert target.exists() or target.is_symlink() + if kind == "symlink": + assert (victim / "owner").read_text() == "safe" + if kind == "extra": + assert (target / "unrelated").read_text() == "preserve" + + +def test_concurrent_creation_is_not_replaced(tmp_path, monkeypatch): + service = ImportService() + options = dataclasses.replace(source_options(tmp_path), set_focus=False) + target = DatasetImportBackend.target_path(options.snapshot_file, options.output_dir) + real = service._dataset_backend._move + + def race(source, dest, identity): + if dest == target: + dest.mkdir() + (dest / "racing-owner").write_text("safe") + real(source, dest, identity) + + monkeypatch.setattr(service._dataset_backend, "_move", race) + with pytest.raises(ImportExecutionError): + service.import_snapshot(options) + assert (target / "racing-owner").read_text() == "safe" + assert list(target.parent.iterdir()) == [target] + + +@pytest.mark.parametrize( + "capacity,fmt,device", + [ + (0, None, None), + (True, None, None), + (2, "single-db", None), + (None, NATIVE_FORMAT, None), + (2, "compatibility-v2", None), + (2, "inline", None), + (2, "unknown", None), + (2, None, -1), + (2, None, True), + ], +) +def test_invalid_options_precede_loading(tmp_path, capacity, fmt, device): + with pytest.raises(InvalidParameterError): + ImportService().import_snapshot( + ImportOptions( + tmp_path / "missing.pkl", events_per_slice=capacity, format=fmt, device=device + ) + ) + + +def test_default_single_db_unchanged_and_separate_target(tmp_path): + options = source_options(tmp_path) + service = ImportService() + single = service.import_snapshot( + dataclasses.replace(options, events_per_slice=None, set_focus=False) + ) + native = service.import_snapshot(dataclasses.replace(options, set_focus=False)) + assert single.db_path.is_file() and single.dataset_path is None + assert single.format == "single-db" and native.db_path.is_dir() + assert service.import_snapshot( + dataclasses.replace(options, events_per_slice=None, set_focus=False) + ).reused diff --git a/tests/core/test_dataset_import_failures.py b/tests/core/test_dataset_import_failures.py new file mode 100644 index 0000000..852c895 --- /dev/null +++ b/tests/core/test_dataset_import_failures.py @@ -0,0 +1,234 @@ +import dataclasses +import os +from pathlib import Path + +import pytest + +from pt_snap_cli.core import sharded_replay_service +from pt_snap_cli.core.errors import ImportExecutionError, SourceChangedError +from pt_snap_cli.core.import_service import ImportService +from pt_snap_cli.core.split_service import SplitService +from tests.core.test_dataset_import import assert_preserved, hashes, saved_state, source_options + + +@pytest.fixture(autouse=True) +def isolate(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + monkeypatch.setenv("HOME", str(tmp_path)) + monkeypatch.setattr(Path, "home", lambda: tmp_path) + monkeypatch.delenv("PT_SNAP_DB_PATH", raising=False) + + +@pytest.mark.parametrize("fail", [False, True]) +def test_directory_descriptors_closed_on_success_and_failure(tmp_path, monkeypatch, fail): + service, options, first, old, focus = saved_state(tmp_path) + descriptors = [] + original = SplitService._stage_identity + + def track(path): + result = original(path) + if result[2] is not None: + descriptors.append(result[2]) + return result + + monkeypatch.setattr(SplitService, "_stage_identity", track) + if fail: + monkeypatch.setattr( + service._dataset_backend.replay, + "stage", + lambda *a, **k: (_ for _ in ()).throw(OSError("load failure")), + ) + with pytest.raises(ImportExecutionError): + service.import_snapshot(options) + assert_preserved(tmp_path, first, old, focus) + else: + service.import_snapshot(options) + assert descriptors + for fd in descriptors: + with pytest.raises(OSError): + os.fstat(fd) + + +def test_actual_source_mutation_during_load_rejects_before_publish(tmp_path, monkeypatch): + service, options, first, old, focus = saved_state(tmp_path) + original = sharded_replay_service.load_snapshot_representation + + def changed(path): + data = original(path) + path.write_bytes(path.read_bytes() + b"actual concurrent source change") + return data + + monkeypatch.setattr(sharded_replay_service, "load_snapshot_representation", changed) + with pytest.raises(SourceChangedError): + service.import_snapshot(options) + assert_preserved(tmp_path, first, old, focus) + + +def test_existing_target_changed_during_staging_is_not_adopted(tmp_path, monkeypatch): + service, options, first, old, focus = saved_state(tmp_path) + original = service._dataset_backend.replay.stage + + def changed(*a, **k): + result = original(*a, **k) + (first.db_path / "racing-owner").write_text("preserve") + return result + + monkeypatch.setattr(service._dataset_backend.replay, "stage", changed) + with pytest.raises(ImportExecutionError, match="preserved"): + service.import_snapshot(options) + assert (first.db_path / "racing-owner").read_text() == "preserve" + current = hashes(first.db_path) + current.pop("racing-owner") + assert current == old + assert (tmp_path / ".pt-snap/focus.json").read_bytes() == focus + assert list(options.output_dir.iterdir()) == [first.db_path] + + +def test_substituted_stage_is_not_deleted_or_followed(tmp_path, monkeypatch): + options = dataclasses.replace(source_options(tmp_path), set_focus=False) + service = ImportService() + original = service._dataset_backend.replay.stage + stages = [] + + def substituted(*a, **k): + result = original(*a, **k) + stage = result.directory.parent + stage.rename(tmp_path / "displaced-owned-stage") + stage.mkdir() + (stage / "unknown-owner").write_text("preserve") + stages.append(stage) + return result + + monkeypatch.setattr(service._dataset_backend.replay, "stage", substituted) + with pytest.raises(ImportExecutionError, match="cleanup failed.*preserved"): + service.import_snapshot(options) + assert (stages[0] / "unknown-owner").read_text() == "preserve" + assert not list(options.output_dir.glob("*.pt-snap-native-v2")) + + +def test_committed_recovery_cleanup_failure_preserves_old_bytes_reports_truthfully( + tmp_path, monkeypatch +): + service, options, first, old, _ = saved_state(tmp_path) + original = service._dataset_backend._remove + + def failed(path, identity): + if path.name == "previous": + raise OSError("cleanup injected failure") + original(path, identity) + + monkeypatch.setattr(service._dataset_backend, "_remove", failed) + with pytest.raises(ImportExecutionError, match="publication committed.*preserved evidence"): + service.import_snapshot(options) + backups = list(options.output_dir.glob("*.recovery-*/previous")) + assert len(backups) == 1 and hashes(backups[0]) == old + assert first.db_path.exists() # Do not falsely report rollback after commit. + + +@pytest.mark.parametrize("atomic_fault", [False, True]) +def test_real_focus_write_mutation_before_error_and_late_restore_error( + tmp_path, monkeypatch, atomic_fault +): + service, options, first, old, focus = saved_state(tmp_path) + options = dataclasses.replace(options, device=0) + path = tmp_path / ".pt-snap/focus.json" + if atomic_fault: + original = os.replace + + def fail_after_rename(source, destination): + original(source, destination) + if Path(destination) == path: + raise OSError("focus rename completed then failed") + + monkeypatch.setattr(os, "replace", fail_after_rename) + else: + original = service._focus_service._config.write_project_focus + + def fail_after_write(*a, **k): + original(*a, **k) + raise OSError("focus write completed then failed") + + monkeypatch.setattr(service._focus_service._config, "write_project_focus", fail_after_write) + with pytest.raises(ImportExecutionError, match="focus .*completed then failed") as error: + service.import_snapshot(options) + assert "rollback also failed" not in str(error.value) + assert_preserved(tmp_path, first, old, focus) + + +def test_parent_finding_substituted_recovery_is_preserved_and_reported(tmp_path, monkeypatch): + service, options, first, old, focus = saved_state(tmp_path) + original = service._dataset_backend._move + displaced = tmp_path / "safe-displaced-previous" + markers = [] + + def fault(source, dest, identity): + if source.name == "artifact": + previous = next(options.output_dir.glob("*.recovery-*/previous")) + assert previous.resolve() == previous and not previous.is_symlink() + assert displaced.resolve() == displaced and not displaced.exists() + previous.rename(displaced) + previous.mkdir() + marker = previous / "foreign-owner" + marker.write_text("must survive") + markers.append(marker) + raise OSError("publication failed after recovery substitution") + original(source, dest, identity) + + monkeypatch.setattr(service._dataset_backend, "_move", fault) + with pytest.raises( + ImportExecutionError, match="directory rollback also failed.*Recovery directory" + ): + service.import_snapshot(options) + assert hashes(displaced) == old + assert markers[0].read_text() == "must survive" + assert (tmp_path / ".pt-snap/focus.json").read_bytes() == focus + assert list(options.output_dir.glob("*.stage-*")) + + +def test_parent_finding_same_inode_focus_checkpoint_is_not_trusted(tmp_path, monkeypatch): + service, options, first, old, _ = saved_state(tmp_path) + original = service._focus_service._config.write_project_focus + checkpoints = [] + + def fault(*a, **k): + original(*a, **k) + backup = next((tmp_path / ".pt-snap").glob("*.recovery")) + inode = backup.stat().st_ino + backup.write_bytes(b"same inode, foreign checkpoint bytes") + assert backup.stat().st_ino == inode + checkpoints.append(backup) + raise OSError("focus failed after checkpoint corruption") + + monkeypatch.setattr(service._focus_service._config, "write_project_focus", fault) + with pytest.raises( + ImportExecutionError, match="focus rollback also failed.*checkpoint content changed" + ): + service.import_snapshot(dataclasses.replace(options, device=0)) + assert hashes(first.db_path) == old + assert checkpoints[0].read_bytes() == b"same inode, foreign checkpoint bytes" + assert (tmp_path / ".pt-snap/focus.json").read_bytes() != checkpoints[0].read_bytes() + + +@pytest.mark.parametrize("bad", ["empty", "corrupt", "absent-device", "alias-parent"]) +def test_dataset_input_and_output_prerequisites(tmp_path, bad): + options = source_options(tmp_path) + if bad == "empty": + from tests.snapshot.test_sharded_replay import write_source + + options = dataclasses.replace( + options, snapshot_file=write_source(tmp_path, {"segments": [], "device_traces": [[]]}) + ) + elif bad == "corrupt": + options.snapshot_file.write_bytes(b"not pickle") + elif bad == "absent-device": + options = dataclasses.replace(options, device=9) + elif bad == "alias-parent": + victim = tmp_path / "other-owner" + victim.mkdir() + alias = tmp_path / "alias" + alias.symlink_to(victim, target_is_directory=True) + options = dataclasses.replace(options, output_dir=alias / "nested") + with pytest.raises(ImportExecutionError): + ImportService().import_snapshot(options) + if bad == "alias-parent": + assert list(victim.iterdir()) == [] diff --git a/tests/core/test_msinsight_export.py b/tests/core/test_msinsight_export.py new file mode 100644 index 0000000..4cd810c --- /dev/null +++ b/tests/core/test_msinsight_export.py @@ -0,0 +1,368 @@ +"""Fixed interface contracts; GUI remains deferred, no vendor code execution.""" + +import copy +import dataclasses +import hashlib +import json +import sqlite3 +from contextlib import closing +from pathlib import Path + +import pytest +from typer.testing import CliRunner + +from pt_snap_cli.api import SnapshotAnalyzer +from pt_snap_cli.cli import _safe_call, app +from pt_snap_cli.core.dataset_contract import ( + ACTION_NAMES, + BLOCK_COLUMNS, + BLOCK_STATES, + DICTIONARY_COLUMNS, + MSINSIGHT_REVISION, + TRACE_COLUMNS, + validate_dataset, +) +from pt_snap_cli.core.dataset_resolver import DatasetResolver +from pt_snap_cli.core.errors import ImportExecutionError, SourceChangedError +from pt_snap_cli.core.focus_service import FocusService +from pt_snap_cli.core.import_service import ImportService +from pt_snap_cli.core.models import ImportOptions +from tests.core.test_dataset_import import hashes +from tests.snapshot.test_sharded_replay import lifecycle_data, write_source + + +@pytest.fixture(autouse=True) +def isolate(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + monkeypatch.setenv("HOME", str(tmp_path)) + monkeypatch.setattr(Path, "home", lambda: tmp_path) + monkeypatch.delenv("PT_SNAP_DB_PATH", raising=False) + + +def options(tmp_path, *, multiple=False, capacity=2, focus=True): + data = lifecycle_data() + data["device_traces"][0] = [e for e in data["device_traces"][0] if e["action"] != "oom"] + if multiple: + data["device_traces"] += [copy.deepcopy(data["device_traces"][0]), []] + data["segments"] += [{**copy.deepcopy(s), "device": 1} for s in data["segments"]] + return ImportOptions( + write_source(tmp_path, data), + tmp_path / "output", + set_focus=focus, + events_per_slice=capacity, + format="msinsight", + ) + + +def rows(path, sql): + with closing(sqlite3.connect(path.as_uri() + "?mode=ro&immutable=1", uri=True)) as conn: + return conn.execute(sql).fetchall() + + +@pytest.mark.parametrize("capacity", [1, 2, 100, None]) +@pytest.mark.parametrize("fmt", ["msinsight", "compatibility-v1"]) +def test_physical_schema_inline_events_streams_stable_references_and_native_parity( + tmp_path, capacity, fmt +): + request = dataclasses.replace(options(tmp_path, multiple=True, capacity=capacity), format=fmt) + service = ImportService() + result = service.import_snapshot(request) + assert result.format == "compatibility-v1" and result.metadata.import_format_version == 1 + assert result.devices == (0, 1) and result.omitted_devices == (2,) + assert result.focus_state.callstack_layout == "v1" + manifest = json.loads((result.db_path / "manifest.json").read_text()) + assert manifest["sourceFile"] == str(request.snapshot_file.resolve()) + assert manifest["eventsPerSlice"] == (capacity or 500000) + raw = request.snapshot_file.read_bytes() + assert manifest["cacheHash"] == hashlib.sha256(b"mem_snapshot_parser_v2" + raw).hexdigest() + assert result.metadata.source_sha256 == hashlib.sha256(raw).hexdigest() != manifest["cacheHash"] + assert manifest["ptSnap"]["identity"]["targetRevision"] == MSINSIGHT_REVISION + native = service.import_snapshot( + dataclasses.replace( + request, format=None, events_per_slice=capacity or 500000, set_focus=False + ) + ) + compatible = DatasetResolver().inspect(result.db_path) + native_dataset = DatasetResolver().inspect(native.db_path) + assert validate_dataset(result.db_path).real_event_count == 10 + for device in (0, 1): + lifetimes = {} + for shard, native_shard in zip( + compatible.device(device).slices, native_dataset.device(device).slices, strict=True + ): + path = result.db_path / shard.file + for table, columns in ( + (f"trace_entry_{device}", TRACE_COLUMNS), + (f"block_{device}", BLOCK_COLUMNS), + ("dictionary", DICTIONARY_COLUMNS), + ): + schema = rows(path, f"PRAGMA table_info({table})") + assert tuple((r[1], r[2]) for r in schema) == columns + assert rows(path, f"SELECT type FROM sqlite_master WHERE name='{table}'") == [ + ("table",) + ] + assert not rows(path, "SELECT name FROM sqlite_master WHERE lower(name)='callstack'") + assert rows(path, "SELECT import_format_version FROM pt_snap_metadata") == [(1,)] + actual = rows(path, f"SELECT * FROM trace_entry_{device} ORDER BY id") + expected = rows( + native.db_path / native_shard.file, + f"SELECT t.id,t.action,t.address,t.size,t.stream,t.allocated,t.active,t.reserved,c.callstack FROM trace_entry_{device} t JOIN callstack c ON c.id=t.callstackId ORDER BY t.id", + ) + assert actual == expected + assert [r[0] for r in actual if r[0] >= 0] == list( + range(shard.start_event_id, shard.end_event_id + 1) + ) + assert set( + rows(path, 'SELECT "key","value" FROM dictionary WHERE "column"=\'action\'') + ) == {(str(k), v) for k, v in enumerate(ACTION_NAMES)} + assert set( + rows(path, 'SELECT "key","value" FROM dictionary WHERE "column"=\'state\'') + ) == {(str(k), v) for k, v in BLOCK_STATES} + references = {r[0]: r[1:] for r in rows(path, "SELECT * FROM pt_snap_block_reference")} + for block in rows(path, f"SELECT * FROM block_{device}"): + signature = block[1:4] + block[5:] + assert block[0] not in lifetimes or lifetimes[block[0]] == signature + lifetimes[block[0]] = signature + if block[6] >= 0: + assert block[6] == 3 # free_completed, NOT free_requested=2 + assert references[block[0]][2] == "source.py:4 free_completed" + before = hashes(result.db_path) + with ( + SnapshotAnalyzer(result.db_path) as analyzer, + SnapshotAnalyzer(native.db_path) as native_analyzer, + ): + assert analyzer.get_focus().callstack_layout == "v1" + for device in (0, 1): + for event in range(5): + assert ( + analyzer.execute_query("event", {"id": event}, device_id=device)["rows"] + == native_analyzer.execute_query("event", {"id": event}, device_id=device)[ + "rows" + ] + ) + assert not hasattr(analyzer, "import_snapshot") + assert hashes(result.db_path) == before + + +def test_identical_complete_cache_reuses_without_loading_even_force(tmp_path, monkeypatch): + request = options(tmp_path, focus=False) + service = ImportService() + first = service.import_snapshot(request) + before = hashes(first.db_path) + monkeypatch.setattr( + service._msinsight_backend.replay, + "stage", + lambda *a, **k: pytest.fail("reused cache loaded pickle"), + ) + for force in (False, True): + reused = service.import_snapshot(dataclasses.replace(request, force=force)) + assert reused.reused and reused.metadata == first.metadata + assert hashes(first.db_path) == before + assert not (tmp_path / ".pt-snap").exists() + + +@pytest.mark.parametrize( + "axis", + [ + "source", + "capacity", + "device", + "sourceFile", + "hash", + "version", + "metadata", + "last-member", + "extra", + "missing", + ], +) +@pytest.mark.parametrize("force", [False, True]) +def test_nonidentical_existing_output_never_replaced(tmp_path, axis, force): + service = ImportService() + request = options(tmp_path) + first = service.import_snapshot(request) + focus = (tmp_path / ".pt-snap/focus.json").read_bytes() + manifest_path = first.db_path / "manifest.json" + data = json.loads(manifest_path.read_text()) + if axis == "source": + request.snapshot_file.write_bytes( + request.snapshot_file.read_bytes() + b"trailing-source-bytes" + ) + elif axis == "capacity": + request = dataclasses.replace(request, events_per_slice=3) + elif axis == "device": + request = dataclasses.replace(request, device=0) + elif axis == "sourceFile": + data["sourceFile"] = "/different/input.pkl" + elif axis == "hash": + data["cacheHash"] = "f" * 64 + elif axis == "version": + data["ptSnap"]["identity"]["exportContractVersion"] += 1 + elif axis == "metadata": + data["ptSnap"]["metadata"]["import_format_version"] = 2 + elif axis == "last-member": + (first.db_path / "device_0/slice_00002.db").write_bytes(b"broken") + elif axis == "extra": + (first.db_path / "user-notes").write_text("preserve") + else: + # Test-owned member removal, never an external artifact. + (first.db_path / "device_0/slice_00002.db").unlink() + if axis in ("sourceFile", "hash", "version", "metadata"): + manifest_path.write_text(json.dumps(data)) + before = hashes(first.db_path) + with pytest.raises(ImportExecutionError, match="preserved even under --force"): + service.import_snapshot(dataclasses.replace(request, force=force)) + assert hashes(first.db_path) == before + assert (tmp_path / ".pt-snap/focus.json").read_bytes() == focus + + +@pytest.mark.parametrize("kind", ["empty", "unknown", "external", "file", "symlink", "dangling"]) +def test_original_external_user_targets_preserved_under_force(tmp_path, kind): + request = options(tmp_path) + target = request.output_dir / "input.pkl.msinsight" + target.parent.mkdir() + if kind in ("symlink", "dangling"): + victim = tmp_path / "victim" + if kind == "symlink": + victim.mkdir() + (victim / "owner").write_text("safe") + target.symlink_to(victim, target_is_directory=True) + elif kind == "file": + target.write_bytes(b"user bytes") + else: + target.mkdir() + if kind == "external": + # A genuine valid base artifact still lacks pt-snap ownership. + from tests.test_dataset_focus import make_dataset + + external = make_dataset(tmp_path / "external", devices=(0,), slices=1) + import shutil + + shutil.copytree(external, target, dirs_exist_ok=True) + if kind != "empty": + (target / "owner").write_text("safe") + before = ( + hashes(target) if target.is_dir() else target.read_bytes() if target.is_file() else None + ) + with pytest.raises(ImportExecutionError): + ImportService().import_snapshot(dataclasses.replace(request, force=True)) + assert target.exists() or target.is_symlink() + if target.is_dir(): + assert hashes(target) == before + if kind == "file": + assert target.read_bytes() == before + if kind == "symlink": + assert (victim / "owner").read_text() == "safe" + assert not (tmp_path / ".pt-snap").exists() + + +@pytest.mark.parametrize( + "kind", ["nonzero", "sparse", "oom", "unknown", "empty", "static-only", "empty-selection"] +) +def test_unsupported_input_fails_without_publication_or_focus(tmp_path, kind): + request = options(tmp_path) + data = lifecycle_data() + data["device_traces"][0] = [e for e in data["device_traces"][0] if e["action"] != "oom"] + if kind in ("nonzero", "sparse"): + for index, event in enumerate(data["device_traces"][0]): + event["id"] = index + 1 if kind == "nonzero" else index * 2 + elif kind in ("oom", "unknown"): + data["device_traces"][0][0]["action"] = "oom" if kind == "oom" else "unknown" + elif kind == "empty": + data = {"segments": [], "device_traces": [[]]} + elif kind == "static-only": + data["device_traces"] = [[]] + else: + data["device_traces"].append([]) + request = dataclasses.replace(request, device=1) + write_source(tmp_path, data) + with pytest.raises(ImportExecutionError): + ImportService().import_snapshot(request) + assert not list(request.output_dir.iterdir()) + assert not (tmp_path / ".pt-snap").exists() + + +@pytest.mark.parametrize( + "phase", ["conversion", "metadata", "validation", "source", "focus", "rename", "race"] +) +def test_failures_preserve_old_focus_and_unknown_owner(tmp_path, monkeypatch, phase): + request = options(tmp_path) + old = tmp_path / ".pt-snap/focus.json" + old.parent.mkdir() + old.write_bytes(b'{"db_path":"/old/path","device_id":0}') + before = old.read_bytes() + service = ImportService() + target = request.output_dir / "input.pkl.msinsight" + + def fail(*a, **k): + raise OSError("injected " + phase) + + if phase == "conversion": + monkeypatch.setattr("pt_snap_cli.core.import_service.convert_shards", fail) + elif phase == "metadata": + monkeypatch.setattr(service._metadata_service, "write", fail) + elif phase == "validation": + monkeypatch.setattr("pt_snap_cli.core.import_service.inspect_owned_cache", fail) + elif phase == "source": + monkeypatch.setattr( + service, + "_recheck_source", + lambda *a: (_ for _ in ()).throw(SourceChangedError("source changed")), + ) + elif phase == "focus": + fs = FocusService() + service._focus_service = fs + + def mutate(*a, **k): + old.write_bytes(b"changed") + fail() + + monkeypatch.setattr(fs._config, "write_project_focus", mutate) + else: + move = service._msinsight_backend._move + + def fault(source, destination, identity): + if phase == "race": + destination.mkdir() + (destination / "owner").write_text("safe") + else: + move(source, destination, identity) + fail() + move(source, destination, identity) + + monkeypatch.setattr(service._msinsight_backend, "_move", fault) + with pytest.raises(ImportExecutionError): + service.import_snapshot(request) + assert old.read_bytes() == before + if phase == "race": + assert (target / "owner").read_text() == "safe" + else: + assert not target.exists() + + +def test_cli_json_explicit_mode_default_adjacency_and_api_focus_parity( + tmp_path, monkeypatch, capsys +): + request = options(tmp_path, capacity=None) + args = ["import", str(request.snapshot_file), "--format", "msinsight", "--json"] + runner = CliRunner() + result = runner.invoke(app, args) + assert result.exit_code == 0, result.output + data = json.loads(result.output) + assert data["format"] == "compatibility-v1" and data["slice_count"] == 1 + assert data["db_path"] == str(request.snapshot_file) + ".msinsight" + with SnapshotAnalyzer(data["db_path"]) as analyzer: + assert ( + analyzer.get_focus().callstack_layout == data["focus_state"]["callstack_layout"] == "v1" + ) + cli = runner.invoke( + app, + ["query", data["db_path"], "--template-use", "event", "--params", '{"id":3}', "--json"], + ) + assert cli.exit_code == 0 + assert json.loads(cli.output)["rows"] == analyzer.execute_query("event", {"id": 3})["rows"] + reused = runner.invoke(app, [*args, "--force", "--no-focus"]) + assert reused.exit_code == 0 and json.loads(reused.output)["reused"] + monkeypatch.setattr("sys.argv", ["pt-snap", *args, "--events-per-slice", "0"]) + assert _safe_call() != 0 + assert json.loads(capsys.readouterr().err)["error"]["code"] == "INVALID_PARAMETER" diff --git a/tests/core/test_sharded_replay_service.py b/tests/core/test_sharded_replay_service.py new file mode 100644 index 0000000..3965f9b --- /dev/null +++ b/tests/core/test_sharded_replay_service.py @@ -0,0 +1,99 @@ +import copy + +import pytest + +from pt_snap_cli.core import sharded_replay_service +from pt_snap_cli.core.sharded_replay_service import ShardedReplayService +from pt_snap_cli.snapshot.tools.adaptors import sharded_replay + +from ..snapshot.test_sharded_replay import lifecycle_data, write_source + + +def test_source_loaded_once_and_one_simulator_per_selected_device(tmp_path, monkeypatch): + data = lifecycle_data() + data["device_traces"].append(copy.deepcopy(data["device_traces"][0])) + data["segments"] += [{**copy.deepcopy(segment), "device": 1} for segment in data["segments"]] + source = write_source(tmp_path, data) + loads, devices = [], [] + load = sharded_replay_service.load_snapshot_representation + simulator = sharded_replay.SimulateDeviceSnapshot + + def counted_load(path): + loads.append(path) + return load(path) + + def counted_simulator(data, device, **kwargs): + devices.append(device) + return simulator(data, device, **kwargs) + + monkeypatch.setattr(sharded_replay_service, "load_snapshot_representation", counted_load) + monkeypatch.setattr(sharded_replay, "SimulateDeviceSnapshot", counted_simulator) + result = ShardedReplayService().stage(source, tmp_path / "stage", events_per_slice=1) + assert loads == [source] + assert devices == [0, 1] + assert result.layout == "native-v2" and result.omitted_devices == () + assert len(result.slices) == 18 + + +@pytest.mark.parametrize("capacity", [0, -1, True, 1.5]) +def test_invalid_capacity_before_load_or_staging(tmp_path, capacity): + with pytest.raises(ValueError, match="events_per_slice"): + ShardedReplayService().stage( + tmp_path / "missing", tmp_path / "stage", events_per_slice=capacity + ) + assert not (tmp_path / "stage").exists() + + +@pytest.mark.parametrize("device", [-1, True, 1.5, 5]) +def test_invalid_or_missing_device_never_creates_stage(tmp_path, device): + source = write_source(tmp_path, lifecycle_data()) + with pytest.raises(ValueError): + ShardedReplayService().stage(source, tmp_path / "stage", events_per_slice=2, device=device) + assert not (tmp_path / "stage").exists() + + +def test_omitted_empty_devices_disclosed(tmp_path): + data = lifecycle_data() + data["device_traces"].append([]) + result = ShardedReplayService().stage( + write_source(tmp_path, data), tmp_path / "stage", events_per_slice=2, device=0 + ) + assert result.omitted_devices == (1,) + + +def test_empty_static_only_and_existing_stage_rejected(tmp_path): + source = write_source(tmp_path, {"segments": [], "device_traces": [[]]}) + with pytest.raises(ValueError, match="trace events"): + ShardedReplayService().stage(source, tmp_path / "stage", events_per_slice=2) + existing = tmp_path / "existing" + existing.mkdir() + marker = existing / "owner.txt" + marker.write_text("preserve") + with pytest.raises(FileExistsError): + ShardedReplayService().stage(source, existing, events_per_slice=2) + assert marker.read_text() == "preserve" + + +def test_symlink_and_dangling_stage_rejected(tmp_path): + (tmp_path / "actual").mkdir() + for name, target in [("link", "actual"), ("dangling", "missing")]: + (tmp_path / name).symlink_to(target) + with pytest.raises(ValueError, match="non-symlink"): + ShardedReplayService().stage(tmp_path / "input", tmp_path / name, events_per_slice=1) + with pytest.raises(ValueError, match="non-symlink"): + ShardedReplayService().stage( + tmp_path / "input", tmp_path / "link" / "child", events_per_slice=1 + ) + + +@pytest.mark.parametrize("ids", [[0, 0], [1, 0], [-1, 0]]) +def test_invalid_native_ids_rejected_without_ready_result(tmp_path, ids): + data = { + "segments": [], + "device_traces": [[{"id": value, "action": "oom", "frames": []} for value in ids]], + } + with pytest.raises(ValueError, match="event IDs"): + ShardedReplayService().stage( + write_source(tmp_path, data), tmp_path / "stage", events_per_slice=1 + ) + assert not list((tmp_path / "stage").iterdir()) diff --git a/tests/query/test_callstack_schema_compat.py b/tests/query/test_callstack_schema_compat.py index 2cbfa53..9363e7a 100644 --- a/tests/query/test_callstack_schema_compat.py +++ b/tests/query/test_callstack_schema_compat.py @@ -368,16 +368,17 @@ def test_v1_callstack_analysis_pagination_and_total(v1_db: Path) -> None: assert [row["callstack"] for row in result.rows] == ["after.py:40", "free.py:30"] -def test_v2_callstack_analysis_keeps_id_aggregation(v2_db: Path) -> None: +def test_v2_callstack_analysis_v2_groups_full_text_and_real_actions(v2_db: Path) -> None: context = Context(v2_db) sql = QueryExecutor(context).render( get_query("callstack_analysis"), {"min_count": 1, "min_size": 0, "limit": -1}, device_id=0, ) - normalized = sql.replace("GROUP BY callstackId", "") - assert "GROUP BY callstackId" in sql - assert "GROUP BY callstack" not in normalized + assert "GROUP BY c.callstack" in sql + assert "t.id >= 0" in sql + assert "GROUP BY callstackId" not in sql + assert get_query("callstack_analysis").semantics_version == 2 context.close() diff --git a/tests/query/test_registry.py b/tests/query/test_registry.py index 23c7646..3a5994b 100644 --- a/tests/query/test_registry.py +++ b/tests/query/test_registry.py @@ -230,7 +230,8 @@ def test_block_template_schema_includes_every_selected_column(self): info = get_template_info("block") assert info is not None - assert [column["column"] for column in info["output_schema"]] == [ + columns = info["output_schema"] + assert [column["column"] for column in columns[:7]] == [ "id", "address", "size", @@ -239,6 +240,22 @@ def test_block_template_schema_includes_every_selected_column(self): "allocEventId", "freeEventId", ] + assert [column["column"] for column in columns[7:]] == [ + "lifecycle_id", + "identity_status", + "allocation_source", + "allocation_source_status", + "free_source", + "free_source_status", + "observation_slice_index", + "state_scope", + "terminal_survivor", + "category", + ] + assert all( + "Dataset only, optional" in " ".join(column["interpretation_limits"]) + for column in columns[7:] + ) def test_packaged_sql_fragment_parameters_declare_choices(self): """Regression for issue #120. diff --git a/tests/query/test_template_semantics.py b/tests/query/test_template_semantics.py index ed4931a..b25c368 100644 --- a/tests/query/test_template_semantics.py +++ b/tests/query/test_template_semantics.py @@ -61,7 +61,7 @@ def test_annotated_templates_share_a_json_serializable_contract() -> None: template = get_query(name) assert template is not None assert template.semantics_version == ( - 2 if name == "active_memory_callstack_at_event" else 1 + 2 if name in ("active_memory_callstack_at_event", "memory_peak", "allocator_gap") else 1 ) assert template.interpretation_limits info = get_template_info(name) diff --git a/tests/skills/test_dataset_scope_contract.py b/tests/skills/test_dataset_scope_contract.py new file mode 100644 index 0000000..19f0f91 --- /dev/null +++ b/tests/skills/test_dataset_scope_contract.py @@ -0,0 +1,54 @@ +"""Shipped instructions: static contracts plus owning executable SQLite/CLI tests. + +These grades do not execute a live model, NPU, original producer or GUI. +""" + +from pathlib import Path + +import pytest + +SKILLS = ( + "pt-snap-helper", + "pt-snap-memory-leak", + "pt-snap-memory-peak-breakdown", + "pt-snap-memory-fragmentation", +) + + +@pytest.mark.parametrize("name", SKILLS) +def test_dataset_scope_identity_and_provenance_are_shipped_identically(name): + authored = Path("skills") / name / "SKILL.md" + bundled = Path("src/pt_snap_cli/bundled_skills") / name / "SKILL.md" + assert authored.read_bytes() == bundled.read_bytes() + text = authored.read_text() + for field in ( + "manifest.json", + "compatibility-v1", + "pt-snap-native-v2", + "fingerprint", + "dataset_support.supported", + "metadata_missing", + "lifecycle_id", + "allocation_source", + "free_source", + "text_kind", + "source_stack_id", + "ordered_frames_complete", + "range_complete", + ): + assert field in text + assert "text-only" in text and "100000" in text and "RSS ceiling" in text + assert "arbitrary" in text and "sibling" in text + assert "ALL" in text and "allocation-only" in text + assert "unknown reason" in text and "invalid metadata" in text + + +def test_updated_peak_capability_keeps_conservative_range_workflow_and_guards(): + text = Path("skills/pt-snap-memory-peak-breakdown/SKILL.md").read_text() + branch = text.split("## Event-Range Workflow")[1].split("## Interpretation Rules")[0] + assert "--start-id" in branch and "--end-id" in branch + assert "Never fabricate an event ID" in branch + assert "NULL" in branch and "empty range" in branch + assert "--template-use memory_peak" in branch + assert "--template-use active_memory_callstack_at_event" in branch + assert "never substitute an unbounded report" in branch diff --git a/tests/skills/test_memory_peak_breakdown_contract.py b/tests/skills/test_memory_peak_breakdown_contract.py index 674f568..c6f6505 100644 --- a/tests/skills/test_memory_peak_breakdown_contract.py +++ b/tests/skills/test_memory_peak_breakdown_contract.py @@ -145,8 +145,13 @@ def test_range_fallback_and_peak_event_semantics_are_explicit() -> None: skill = _skill() normalized = _normalized_skill() - assert "`pt-snap report peak-memory` is full-trace only" in skill - assert "Do not use `report peak-memory` for a bounded range" in skill + assert "without range bounds selects the full real trace" in normalized + assert "supports `--start-id` / `--end-id`" in normalized + assert "never substitute an unbounded report or another event" in normalized + report_help = CliRunner().invoke(app, ["report", "peak-memory", "--help"]) + assert report_help.exit_code == 0 + help_text = re.sub(r"\x1b\[[0-9;]*m", "", report_help.output) + assert "--start-id" in help_text and "--end-id" in help_text assert "--template-use memory_peak --params" in skill assert "--template-use allocator_gap --params" in skill assert '"start_id": , "end_id": ' in skill diff --git a/tests/snapshot/test_sharded_replay.py b/tests/snapshot/test_sharded_replay.py new file mode 100644 index 0000000..f836f43 --- /dev/null +++ b/tests/snapshot/test_sharded_replay.py @@ -0,0 +1,440 @@ +from __future__ import annotations + +import copy +import sqlite3 +from contextlib import closing + +import pytest + +from pt_snap_cli.core.sharded_replay_service import ShardedReplayService +from pt_snap_cli.snapshot.base import Block, BlockState +from pt_snap_cli.snapshot.representation import save_snapshot_representation +from pt_snap_cli.snapshot.simulate import SimulateDeviceSnapshot +from pt_snap_cli.snapshot.tools.adaptors import sharded_replay, snapshot2db +from pt_snap_cli.snapshot.tools.adaptors.database.snapshot_db import SnapshotDb + +from .helpers import FIXTURE_DIR +from .test_simulate import Hooker, make_torch_npu_workspace_snapshot + + +def event(action, addr=1000, size=16, stream=0): + return { + "action": action, + "addr": addr, + "size": size, + "stream": stream, + "frames": [{"filename": "source.py", "line": 4, "name": action}], + } + + +def block(addr, size, requested): + return { + "address": addr, + "size": size, + "requested_size": requested, + "state": "active_allocated", + "frames": [], + } + + +def segment(stream, total, blocks): + return { + "address": 1000, + "total_size": total, + "stream": stream, + "segment_type": "small", + "allocated_size": sum(item["size"] for item in blocks), + "active_size": sum(item["size"] for item in blocks), + "device": 0, + "is_expandable": False, + "frames": [], + "blocks": blocks, + } + + +def lifecycle_data(): + return { + "segments": [ + segment(0, 64, [block(1000, 16, 14), block(1040, 8, 6)]), + segment(1, 32, [block(1000, 8, 5)]), + ], + "device_traces": [ + [ + event("alloc"), + event("alloc", size=8, stream=1), + event("oom"), + event("free_requested"), + event("oom"), + event("free_completed"), + event("alloc"), + event("oom"), + event("oom"), + ] + ], + } + + +def write_source(tmp_path, data): + path = tmp_path / "input.pkl" + save_snapshot_representation(data, path, "pickle") + return path + + +def rows(path, sql): + with closing(sqlite3.connect(path)) as conn: + return conn.execute(sql).fetchall() + + +@pytest.mark.parametrize("capacity", [1, 2, 3, 100]) +def test_shards_equal_single_db_events_blocks_boundaries_and_references(tmp_path, capacity): + data = lifecycle_data() + original = copy.deepcopy(data) + source = write_source(tmp_path, data) + full = tmp_path / "full.db" + assert snapshot2db.dump(source, full) + result = ShardedReplayService().stage(source, tmp_path / "stage", events_per_slice=capacity) + full_events = rows( + full, + "SELECT t.id,t.action,t.address,t.size,t.stream,t.allocated,t.active,t.reserved,c.callstack FROM trace_entry_0 t JOIN callstack c ON c.id=t.callstackId WHERE t.id>=0 ORDER BY t.id", + ) + collected = [] + identities = {} + for item in result.slices: + collected += rows( + item.file, + "SELECT t.id,t.action,t.address,t.size,t.stream,t.allocated,t.active,t.reserved,c.callstack FROM trace_entry_0 t JOIN callstack c ON c.id=t.callstackId WHERE t.id>=0 ORDER BY t.id", + ) + assert len(rows(item.file, "SELECT id FROM trace_entry_0 WHERE id>=0")) <= capacity + references = { + row[0]: row[1:] for row in rows(item.file, "SELECT * FROM pt_snap_block_reference") + } + shard_blocks = rows(item.file, "SELECT * FROM block_0") + for record in shard_blocks: + signature = record[1:4] + record[5:] + assert record[0] not in identities or identities[record[0]] == signature + identities[record[0]] = signature + if record[5] >= 0: + assert record[0] == record[5] + assert references[record[0]][1] == "source.py:4 alloc" + else: + assert record[0] < 0 and references[record[0]][1] is None + if record[6] >= 0: + assert record[6] == 5 # Never free_requested=3. + assert references[record[0]][2] == "source.py:4 free_completed" + # Replay the full simulator to the identical left boundary as an oracle. + oracle = SimulateDeviceSnapshot(data, 0) + assert oracle.replay_until(item.start_position) + state = oracle.device_snapshot + expected_blocks = sorted( + (seg.stream, b.address, b.size, b.requested_size, b.state) + for seg in state.segments + for b in seg.blocks + ) + observed_blocks = sorted( + ( + references[r[0]][0], + r[1], + r[2], + r[3], + {-1: "inactive", 0: "active_pending_free", 1: "active_allocated"}[r[4]], + ) + for r in shard_blocks + if (r[5] == -1 or r[5] < item.start_event_id) + and (r[6] == -1 or r[6] >= item.start_event_id) + ) + assert observed_blocks == expected_blocks + boundaries = rows( + item.file, + "SELECT action,address,size,stream,allocated,active,reserved FROM trace_entry_0 WHERE id<0", + ) + assert sorted(row[1:4] for row in boundaries) == sorted( + (seg.address, seg.total_size, seg.stream) for seg in state.segments + ) + assert all( + row[4:] == (state.total_allocated, state.total_activated, state.total_reserved) + for row in boundaries + ) + # Active geometry at EVERY real event, not just shard edges. + for position in range(item.start_position, item.end_position + 1): + current = SimulateDeviceSnapshot(data, 0) + assert current.replay_until(position + 1) + expected = sorted( + (seg.stream, b.address, b.size, b.requested_size) + for seg in current.device_snapshot.segments + for b in seg.blocks + ) + active = sorted( + (references[r[0]][0], r[1], r[2], r[3]) + for r in shard_blocks + if (r[5] == -1 or r[5] <= position) and (r[6] == -1 or r[6] > position) + ) + assert active == expected + assert collected == full_events + expected_signatures = {row[1:4] + row[5:] for row in rows(full, "SELECT * FROM block_0")} + assert set(identities.values()) == expected_signatures + assert {key for key in identities if key >= 0} == {0, 1, 6} + assert data == original + assert not (result.directory / "manifest.json").exists() + + +def test_pending_free_left_boundary_has_state_zero(tmp_path): + result = ShardedReplayService().stage( + write_source(tmp_path, lifecycle_data()), tmp_path / "stage", events_per_slice=2 + ) + assert rows(result.slices[2].file, "SELECT id,state,freeEventId FROM block_0 WHERE id=0") == [ + (0, 0, 5) + ] + + +@pytest.mark.parametrize( + "fixture", + [ + "snapshot_expandable.pkl", + "snapshot_with_empty_cache_expandable.pkl", + "snapshot_with_multi_devices.pkl", + ], +) +def test_reviewed_fixtures_real_events_match_single_db(tmp_path, fixture): + source = FIXTURE_DIR / fixture + full = tmp_path / "full.db" + assert snapshot2db.dump(source, full) + result = ShardedReplayService().stage(source, tmp_path / "stage", events_per_slice=1000) + for device in {item.device for item in result.slices}: + observed = [] + for item in result.slices: + if item.device == device: + observed += rows( + item.file, + f"SELECT id,action,address,size,stream,allocated,active,reserved FROM trace_entry_{device} WHERE id>=0 ORDER BY id", + ) + assert observed == rows( + full, + f"SELECT id,action,address,size,stream,allocated,active,reserved FROM trace_entry_{device} WHERE id>=0 ORDER BY id", + ) + + +def test_workspace_oom_and_sparse_native_ids_are_not_relabelled(tmp_path): + data = make_torch_npu_workspace_snapshot() + data["device_traces"][0].append( + {"action": "oom", "size": 2048, "device_free": 512, "frames": []} + ) + for index, entry in enumerate(data["device_traces"][0]): + entry["id"] = 10 + index * 2 + source = write_source(tmp_path, data) + result = ShardedReplayService().stage(source, tmp_path / "stage", events_per_slice=1) + actions = [ + rows( + item.file, "SELECT id,action,allocated,active,reserved FROM trace_entry_0 WHERE id>=0" + )[0] + for item in result.slices + ] + assert [row[:2] for row in actions] == [(10, 7), (12, 2), (14, 4), (16, 8)] + assert actions[-1][2:] == (4096, 4096, 4096) + assert rows(result.slices[-1].file, "SELECT id,allocEventId FROM block_0") == [(14, 14)] + + +def test_pause_resume_matches_full_hooks_and_state(): + data = lifecycle_data() + complete = SimulateDeviceSnapshot(data, 0) + paused = SimulateDeviceSnapshot(data, 0) + hooks = [Hooker(), Hooker()] + complete.register_hooker(hooks[0]) + paused.register_hooker(hooks[1]) + assert complete.replay() + for endpoint in [9, 7, 7, 5, 2, 0, 0]: + assert paused.replay_until(endpoint) + assert hooks[0].calls == hooks[1].calls + assert complete.device_snapshot.to_dict() == paused.device_snapshot.to_dict() + + +@pytest.mark.parametrize("endpoint", [-1, True, 1.5, 10]) +def test_invalid_pause_endpoint_preserves_state(endpoint): + simulator = SimulateDeviceSnapshot(lifecycle_data(), 0) + before = simulator.device_snapshot.to_dict() + with pytest.raises(ValueError, match="remaining_events"): + simulator.replay_until(endpoint) + assert simulator.device_snapshot.to_dict() == before + + +@pytest.mark.parametrize("phase", ["pre", "post", "allocator"]) +def test_pause_failure_semantics_remain_original(phase, monkeypatch): + simulator = SimulateDeviceSnapshot(lifecycle_data(), 0) + hook = Hooker(pre=phase != "pre", post=phase != "post") + simulator.register_hooker(hook) + if phase == "allocator": + monkeypatch.setattr(simulator.replay_executor, "execute", lambda _: False) + assert not simulator.replay_until(8) + assert len(simulator.device_snapshot.trace_entries) == (8 if phase == "post" else 9) + + +@pytest.mark.parametrize( + "failure", ["replay", "boundary", "flush", "backfill", "validation", "constructor"] +) +def test_failures_close_every_writer_and_return_no_ready_result(tmp_path, monkeypatch, failure): + connections = [] + original = SnapshotDb.__init__ + + def track(self, *args, **kwargs): + original(self, *args, **kwargs) + connections.append(self.conn) + + monkeypatch.setattr(SnapshotDb, "__init__", track) + + def fail(*args, **kwargs): + raise RuntimeError("injected failure") + + if failure == "replay": + monkeypatch.setattr(SimulateDeviceSnapshot, "replay_until", lambda *_: False) + elif failure == "boundary": + monkeypatch.setattr(sharded_replay._SliceHook, "dump_left_boundary", fail) + elif failure == "flush": + monkeypatch.setattr(snapshot2db.SnapshotDbHandler, "flush", fail) + elif failure == "constructor": + monkeypatch.setattr(SnapshotDb, "create_block_table", fail) + elif failure == "backfill": + monkeypatch.setattr(sharded_replay.BlockRegistry, "finalize", fail) + else: + monkeypatch.setattr(sharded_replay.BlockRegistry, "validate", fail) + with pytest.raises(RuntimeError): + ShardedReplayService().stage( + write_source(tmp_path, lifecycle_data()), tmp_path / "stage", events_per_slice=2 + ) + assert connections + for conn in connections: + with pytest.raises(sqlite3.ProgrammingError, match="closed"): + conn.execute("SELECT 1") + assert not (tmp_path / "stage" / "manifest.json").exists() + + +def test_registry_does_not_merge_distinct_same_address_objects(): + registry = sharded_replay.BlockRegistry() + a = Block(address=1, size=2, requested_size=1, state=BlockState.ACTIVE_ALLOCATED) + b = copy.copy(a) + assert registry.observe(a, 0).token != registry.observe(b, 0).token + assert registry.observe(a, 0).token == -1 + with pytest.raises(RuntimeError, match="stream"): + registry.observe(a, 1) + + +def test_finalization_sql_failure_closes_rw_connection(tmp_path, monkeypatch): + original = sharded_replay.BlockRegistry.finalize + captured = [] + + def broken(self): + path = next(iter(self.members)) + with closing(sqlite3.connect(path)) as conn, conn: + conn.execute("DELETE FROM block_0") + real_connect = sqlite3.connect + + def connect(*args, **kwargs): + conn = real_connect(*args, **kwargs) + captured.append(conn) + return conn + + monkeypatch.setattr(sharded_replay.sqlite3, "connect", connect) + return original(self) + + monkeypatch.setattr(sharded_replay.BlockRegistry, "finalize", broken) + with pytest.raises(RuntimeError, match="Missing provisional"): + ShardedReplayService().stage( + write_source(tmp_path, lifecycle_data()), tmp_path / "stage", events_per_slice=2 + ) + assert captured + for conn in captured: + with pytest.raises(sqlite3.ProgrammingError): + conn.execute("SELECT 1") + + +def test_snapshot_db_constructor_failure_closes_connection(tmp_path, monkeypatch): + captured = [] + + def fail(self): + captured.append(self.conn) + raise RuntimeError("constructor failure") + + monkeypatch.setattr(SnapshotDb, "_clear_old_tables", fail) + with pytest.raises(RuntimeError): + SnapshotDb(str(tmp_path / "failure.db")) + with pytest.raises(sqlite3.ProgrammingError): + captured[0].execute("SELECT 1") + + +def test_writer_commit_failure_closes_connection(tmp_path): + handler = snapshot2db.SnapshotDbHandler(str(tmp_path / "db"), [0]) + conn = handler.db.conn + + class BrokenCommit: + def commit(self): + raise RuntimeError("commit failure") + + def close(self): + conn.close() + + handler.db.conn = BrokenCommit() + with pytest.raises(RuntimeError): + handler.close() + handler.close() + with pytest.raises(sqlite3.ProgrammingError): + conn.execute("SELECT 1") + + +def test_actual_sql_backfill_abort_rolls_back_and_closes(tmp_path, monkeypatch): + original = sharded_replay.BlockRegistry.finalize + real_connect = sqlite3.connect + captured, paths = [], [] + + def rejected(self): + path = next(iter(self.members)) + paths.append(path) + with closing(real_connect(path)) as conn, conn: + conn.execute( + "CREATE TRIGGER reject_patch BEFORE UPDATE ON block_0 " + "BEGIN SELECT RAISE(ABORT, 'backfill rejected'); END" + ) + + def connect(*args, **kwargs): + conn = real_connect(*args, **kwargs) + captured.append(conn) + return conn + + monkeypatch.setattr(sharded_replay.sqlite3, "connect", connect) + original(self) + + monkeypatch.setattr(sharded_replay.BlockRegistry, "finalize", rejected) + with pytest.raises(sqlite3.IntegrityError, match="backfill rejected"): + ShardedReplayService().stage( + write_source(tmp_path, lifecycle_data()), tmp_path / "stage", events_per_slice=2 + ) + for conn in captured: + with pytest.raises(sqlite3.ProgrammingError, match="closed"): + conn.execute("SELECT 1") + with closing(real_connect(paths[0])) as conn: + assert all(row[0] < 0 for row in conn.execute("SELECT id FROM block_0")) + assert not conn.execute( + "SELECT name FROM sqlite_master WHERE name='pt_snap_block_reference'" + ).fetchall() + + +@pytest.mark.parametrize( + ("sql", "message"), + [ + ("DELETE FROM trace_entry_0 WHERE id>=0", "lost or changed real events"), + ("UPDATE block_0 SET requestedSize=0", "identity/lifetime"), + ("DELETE FROM pt_snap_block_reference", "callstack references"), + ], +) +def test_final_validation_rejects_corrupt_finalized_shard(tmp_path, monkeypatch, sql, message): + original = sharded_replay.BlockRegistry.finalize + + def corrupt(self): + original(self) + path = next(iter(self.members)) + with closing(sqlite3.connect(path)) as conn, conn: + conn.execute(sql) + + monkeypatch.setattr(sharded_replay.BlockRegistry, "finalize", corrupt) + with pytest.raises(RuntimeError, match=message): + ShardedReplayService().stage( + write_source(tmp_path, lifecycle_data()), tmp_path / "stage", events_per_slice=2 + ) + assert not (tmp_path / "stage" / "manifest.json").exists() diff --git a/tests/test_contract_cli_api.py b/tests/test_contract_cli_api.py index 9149593..0cf9cfd 100644 --- a/tests/test_contract_cli_api.py +++ b/tests/test_contract_cli_api.py @@ -26,6 +26,7 @@ from pt_snap_cli.core.json_codec import JSON_SCHEMA_VERSION from pt_snap_cli.query.config import QueryParameter, QueryTemplate from pt_snap_cli.query.registry import QueryRegistry, register_query +from tests.test_dataset_focus import make_dataset runner = CliRunner() @@ -82,6 +83,47 @@ def test_stack_summary_cli_api_and_catalog_contract(contract_db: Path, template: assert entry["parameters"] == info["parameters"] +@pytest.mark.parametrize( + "template,params", + [ + ("memory_peak", {}), + ("allocator_gap", {"start_id": 1, "end_id": 4}), + ("event", {"min_id": 1, "max_id": 4, "limit": 2, "offset": 1}), + ("allocation", {"min_id": 1, "max_id": 4, "order_by": "active"}), + ("callstack_analysis", {"min_count": 1, "limit": 1}), + ("block", {}), + ("leak_detection", {}), + ], +) +def test_dataset_global_cli_api_scope_catalog_and_total_contract(tmp_path, template, params): + root = make_dataset(tmp_path / "global-contract", devices=(0,)) + with SnapshotAnalyzer(root) as analyzer: + api = analyzer.execute_query(template, params, exact_total=True) + cli = runner.invoke( + app, + [ + "query", + str(root), + "--template-use", + template, + "--params", + json.dumps(params), + "--exact-total", + "--json", + ], + ) + assert cli.exit_code == 0, cli.output + payload = json.loads(cli.stdout) + for key in api: + assert payload[key] == api[key] + info = analyzer.get_template_info(template) + assert info["dataset_support"]["supported"] + catalog = runner.invoke(app, ["capabilities", "--json"]) + entry = next(c for c in json.loads(catalog.stdout)["templates"] if c["name"] == template) + assert entry["dataset_support"] == info["dataset_support"] + assert api["scope"]["boundary_events_included"] is False + + def create_contract_db(db_path: Path) -> Path: """Create a tiny snapshot database shared by CLI and API contract paths.""" conn = sqlite3.connect(str(db_path)) @@ -236,6 +278,8 @@ def _normalize_cli_template_info(output: str) -> dict[str, object]: info["category"] = stripped.removeprefix("Category: ") elif stripped.startswith("Devices: "): info["devices"] = stripped.removeprefix("Devices: ") + elif stripped.startswith("Dataset Support: "): + info["dataset_support"] = json.loads(stripped.removeprefix("Dataset Support: ")) elif stripped.startswith("Semantics Version: "): text = stripped.removeprefix("Semantics Version: ") info["semantics_version"] = None if text == "none" else int(text) @@ -411,7 +455,7 @@ def test_template_semantics_contract_matches_cli_and_api(contract_db: Path) -> N assert cli_info == api_info json.dumps(api_info) assert api_info["semantics_version"] == ( - 2 if name == "active_memory_callstack_at_event" else 1 + 2 if name in ("active_memory_callstack_at_event", "memory_peak", "allocator_gap") else 1 ) assert api_info["interpretation_limits"] @@ -787,6 +831,191 @@ def test_capabilities_template_matches_template_info_json( assert "empty strings, NULL text or IDs" in " ".join(callstack["interpretation_limits"]) +@pytest.mark.parametrize("use_manifest", [False, True]) +def test_dataset_focus_overview_query_contract(tmp_path: Path, use_manifest: bool) -> None: + root = make_dataset(tmp_path / "dataset") + path = root / "manifest.json" if use_manifest else root + with SnapshotAnalyzer() as analyzer: + state = analyzer.set_focus(str(path), 2) + focus = _json_stdout(runner.invoke(app, ["focus", str(path), "--device", "2", "--json"])) + assert {key: focus[key] for key in _focus_payload(state)} == _focus_payload(state) + overview = _json_stdout(runner.invoke(app, ["overview", str(path), "--json"])) + api_overview = analyzer.get_database_overview() + assert {key: overview[key] for key in api_overview} == api_overview + metadata_result = runner.invoke(app, ["metadata", str(path), "--json"]) + assert metadata_result.exit_code == 0, metadata_result.output + assert json.loads(metadata_result.stdout) == analyzer.get_database_metadata() + for params, selector in [ + ({"id": 4}, []), + ({}, ["--slice", "1"]), + ({"min_id": 2, "max_id": 3}, []), + ]: + cli = _json_stdout( + runner.invoke( + app, + [ + "query", + str(path), + "--device", + "2", + "--template-use", + "event", + "--params", + json.dumps(params), + "--json", + *selector, + ], + ) + ) + api = analyzer.execute_query( + "event", params, device_id=2, slice_index=1 if selector else None + ) + assert {key: cli[key] for key in api} == api + assert cli["scope"]["boundary_events_included"] is False + error = runner.invoke( + app, ["query", str(path), "--template-use", "event", "--params", '{"id": -1}', "--json"] + ) + assert error.exit_code == 1 + assert json.loads(error.stderr)["error"]["code"] == "INVALID_PARAMETER" + global_peak = runner.invoke( + app, ["query", str(path), "--template-use", "memory_peak", "--json"] + ) + assert global_peak.exit_code == 0, global_peak.output + peak_payload = json.loads(global_peak.stdout) + assert peak_payload["scope"]["slice_indices"] == [0, 1, 2] + assert peak_payload["rows"][0]["peak_active"] == 8 # never synthetic 999 + manifest = root / "manifest.json" + data = json.loads(manifest.read_text()) + data["status"] = "building" + manifest.write_text(json.dumps(data)) + error = runner.invoke(app, ["focus", str(path), "--json"]) + assert error.exit_code == 1 + assert json.loads(error.stderr)["error"]["code"] == "DATABASE_SCHEMA_INVALID" + with pytest.raises(ValueError, match="complete"): + analyzer.set_focus(str(path)) + + +def test_compatible_import_focus_metadata_query_contract(tmp_path: Path, monkeypatch) -> None: + from tests.core.test_msinsight_export import options as compatible_options + + monkeypatch.chdir(tmp_path) + monkeypatch.setenv("HOME", str(tmp_path)) + monkeypatch.setattr(Path, "home", lambda: tmp_path) + monkeypatch.delenv("PT_SNAP_DB_PATH", raising=False) + request = compatible_options(tmp_path, multiple=True) + imported = _json_stdout( + runner.invoke( + app, + [ + "import", + str(request.snapshot_file), + "--format", + "msinsight", + "--events-per-slice", + "2", + "--json", + ], + ) + ) + assert imported["format"] == "compatibility-v1" + path = imported["dataset_path"] + focused = _json_stdout(runner.invoke(app, ["focus", path, "--device", "1", "--json"])) + with SnapshotAnalyzer(path, device_id=1) as analyzer: + state = analyzer.get_focus() + assert state.available_devices == focused["available_devices"] == [0, 1] + assert state.callstack_layout == focused["callstack_layout"] == "v1" + metadata_result = runner.invoke(app, ["metadata", path, "--json"]) + assert metadata_result.exit_code == 0, metadata_result.output + metadata = json.loads(metadata_result.output) + assert metadata == analyzer.get_database_metadata() + assert metadata["status"] == "available" + assert metadata["metadata"]["import_format_version"] == 1 + overview_result = runner.invoke(app, ["overview", path, "--json"]) + assert overview_result.exit_code == 0, overview_result.output + overview = json.loads(overview_result.output) + api_overview = analyzer.get_database_overview() + assert {key: overview[key] for key in api_overview} == api_overview + for params, selector in [ + ({"id": 3}, []), + ({}, ["--slice", "1"]), + ({"min_id": 2, "max_id": 3}, []), + ]: + cli = _json_stdout( + runner.invoke( + app, + [ + "query", + path, + "--device", + "1", + "--template-use", + "event", + "--params", + json.dumps(params), + "--json", + *selector, + ], + ) + ) + api = analyzer.execute_query("event", params, slice_index=1 if selector else None) + assert {key: cli[key] for key in api} == api + assert cli["scope"]["boundary_events_included"] is False + assert not hasattr(analyzer, "import_snapshot") + + +@pytest.mark.parametrize( + "template", ["event", "active_blocks_at_event", "active_memory_callstack_at_event"] +) +@pytest.mark.parametrize("frames", [False, True]) +def test_dataset_point_event_sources_cli_api_report_contract( + tmp_path: Path, template: str, frames: bool +) -> None: + from pt_snap_cli.core.report_service import ReportService + from tests.core.test_dataset_attribution import add_frames, case + + root = case(tmp_path) + if frames: + add_frames(root) + params = {"id": 0, "stack_bytes": 0} if template == "event" else {"event_id": 8} + if template == "active_memory_callstack_at_event": + params.update({"top_n": 1, "stack_bytes": 0}) + cli = _json_stdout( + runner.invoke( + app, + [ + "query", + str(root), + "--device", + "2", + "--template-use", + template, + "--params", + json.dumps(params), + "-n", + "1", + "--exact-total", + "--json", + ], + ) + ) + with SnapshotAnalyzer(root, device_id=2) as analyzer: + api = analyzer.execute_query(template, params, max_rows=1, exact_total=True) + assert {key: cli[key] for key in api} == api + assert cli["scope"]["source_coverage"]["range_complete"] + if template != "event": + assert ( + cli["truncated"] and cli["scope"]["source_coverage"]["allocation_source_complete"] + ) + if template == "active_memory_callstack_at_event": + report = ReportService() + try: + shared = report.event_attribution(8, root, 2, limit=1, stack_bytes=0) + assert shared.rows[0] == api["rows"][0] + assert shared.scope == api["scope"] + finally: + report.close() + + def test_overview_error_contract_maps_four_domain_failures(tmp_path: Path) -> None: missing = runner.invoke(app, ["overview", str(tmp_path / "missing.db"), "--json"]) missing_error = json.loads(missing.stderr)["error"] diff --git a/tests/test_dataset_contract.py b/tests/test_dataset_contract.py new file mode 100644 index 0000000..2ceae85 --- /dev/null +++ b/tests/test_dataset_contract.py @@ -0,0 +1,413 @@ +"""Synthetic P0 contract tests; no executable snapshot fixtures or upstream code.""" + +import json +import sqlite3 +from contextlib import closing +from copy import deepcopy +from pathlib import Path + +import pytest + +from pt_snap_cli.context import Context +from pt_snap_cli.core.dataset_contract import ( + ACTION_NAMES, + BLOCK_STATES, + DatasetContractError, + QueryScope, + parse_manifest, + validate_dataset, + validate_event_ids, +) + + +def manifest_data(): + return { + "schemaVersion": 1, + "status": "complete", + "sourceFile": "/capture/snapshot.pkl", + "cacheHash": "producer-specific-opaque-identity", + "eventsPerSlice": 2, + "devices": { + "0": { + "eventCount": 4, + "sliceCount": 2, + "readySlices": [0, 1], + "slices": [ + { + "index": 0, + "startEventId": 0, + "endEventId": 1, + "file": "device_0/slice_00000.db", + "ready": True, + }, + { + "index": 1, + "startEventId": 2, + "endEventId": 3, + "file": "device_0/slice_00001.db", + "ready": True, + }, + ], + } + }, + } + + +def make_dataset(root): + root.mkdir() + data = manifest_data() + (root / "manifest.json").write_text(json.dumps(data), encoding="utf-8") + (root / "device_0").mkdir() + for index in range(2): + path = root / f"device_0/slice_{index:05d}.db" + with closing(sqlite3.connect(path)) as conn, conn: + # Independent fixed DDL catches changes to the validator's constants. + conn.executescript( + 'CREATE TABLE dictionary ("table" TEXT, "column" TEXT, "key" TEXT, "value" TEXT);' + "CREATE TABLE trace_entry_0 (id INTEGER PRIMARY KEY, action INTEGER, " + "address INTEGER, size INTEGER, stream INTEGER, allocated INTEGER, " + "active INTEGER, reserved INTEGER, callstack TEXT);" + "CREATE TABLE block_0 (id INTEGER PRIMARY KEY, address INTEGER, size INTEGER, " + "requestedSize INTEGER, state INTEGER, allocEventId INTEGER, freeEventId INTEGER);" + ) + conn.executemany( + "INSERT INTO dictionary VALUES (?, ?, ?, ?)", + [ + ("trace_entry_0", "action", str(key), value) + for key, value in enumerate(ACTION_NAMES) + ] + + [("block_0", "state", str(key), value) for key, value in BLOCK_STATES], + ) + for event_id in range(index * 2, index * 2 + 2): + conn.execute( + "INSERT INTO trace_entry_0 VALUES (?,4,100,8,0,8,8,16,'file.py:1 fn')", + (event_id,), + ) + # Artificial boundary metrics exceed the real peak; they are not real events. + conn.execute("INSERT INTO trace_entry_0 VALUES (-1,2,100,16,0,999,999,999,'boundary')") + conn.executemany( + "INSERT INTO block_0 VALUES (?,?,?,?,?,?,?)", + [ + (0, 100, 8, 8, 1, 0, -1), + (-1, 200, 8, 4, 1, -1, -1), # Static/preexisting, unknown lifetime. + ], + ) + return root + + +def mutate_db(root, sql): + with closing(sqlite3.connect(root / "device_0/slice_00001.db")) as conn, conn: + conn.executescript(sql) + + +def test_valid_external_dataset_boundary_and_unknown_extensions(tmp_path): + root = make_dataset(tmp_path / "snapshot.pkl.msinsight") + data = manifest_data() + data["extensions"] = {"future.vendor": {"version": 999, "frames": "claimed"}} + (root / "manifest.json").write_text(json.dumps(data), encoding="utf-8") + mutate_db( + root, + "CREATE TABLE pt_snap_future (opaque TEXT); INSERT INTO pt_snap_future VALUES ('ignored');", + ) + before = {p: p.read_bytes() for p in root.rglob("*") if p.is_file()} + result = validate_dataset(root) + assert result.real_event_count == 4 + assert result.boundary_event_count == 2 + assert result.text_callstacks is True + assert result.structured_frames is False + assert result.query_execution is False + assert before == {p: p.read_bytes() for p in before} + assert not list(root.rglob("*-journal")) + context = Context(root / "device_0/slice_00000.db") + try: + assert context.callstack_layout == "v1" + finally: + context.close() + + +def test_address_reuse_is_not_identity_and_state_is_slice_local(tmp_path): + root = make_dataset(tmp_path / "dataset") + mutate_db( + root, + "INSERT INTO block_0 VALUES (2,100,8,8,1,2,-1); " + "INSERT INTO block_0 VALUES (-2,200,8,4,1,-1,-1); " + "UPDATE block_0 SET state=0 WHERE id=0;", + ) + assert validate_dataset(root).real_event_count == 4 + + +@pytest.mark.parametrize("ids", [[1, 2], [0, 2], [0, 0], [0, -1], [], [True], [0, 1.0]]) +def test_producer_rejects_nonzero_sparse_empty_or_invalid_real_ids(ids): + with pytest.raises(DatasetContractError): + validate_event_ids(ids) + + +def test_real_ids_use_actual_positions_not_maximum(): + assert validate_event_ids(iter([0, 1, 2])) == 3 + + +@pytest.mark.parametrize( + "field,value,code", + [ + ("schemaVersion", 2, "version"), + ("schemaVersion", True, "type"), + ("schemaVersion", 2**31, "range"), + ("eventsPerSlice", 0, "range"), + ("eventsPerSlice", 2**63, "range"), + ("eventsPerSlice", 2.0, "type"), + ("sourceFile", "", "range"), + ("cacheHash", None, "type"), + ("status", "failed", "status"), + ("devices", {}, "empty"), + ("devices", [], "type"), + ], +) +def test_manifest_top_level_errors(field, value, code): + data = manifest_data() + data[field] = value + with pytest.raises(DatasetContractError) as exc: + parse_manifest(data) + assert exc.value.code == code + assert exc.value.location == field + + +@pytest.mark.parametrize( + "field,value", + [ + ("eventCount", 0), + ("eventCount", 5), + ("eventCount", True), + ("sliceCount", 1), + ("sliceCount", 0), + ("readySlices", [0, 0]), + ("readySlices", [0]), + ("readySlices", [0, 2]), + ("readySlices", [True, 1]), + ], +) +def test_device_counts_readiness_and_empty_device_rejected(field, value): + data = manifest_data() + data["devices"]["0"][field] = value + with pytest.raises(DatasetContractError, match="devices.0"): + parse_manifest(data) + + +@pytest.mark.parametrize("device", ["01", "-1", "device_0", "1/../0", str(2**31)]) +def test_invalid_device_identity(device): + data = manifest_data() + data["devices"][device] = data["devices"].pop("0") + with pytest.raises(DatasetContractError): + parse_manifest(data) + + +@pytest.mark.parametrize( + "field,value", + [ + ("startEventId", 1), + ("startEventId", 3), + ("endEventId", 4), + ("endEventId", 1), + ("index", 0), + ("index", True), + ("ready", "true"), + ("file", "../other.db"), + ("file", "/other.db"), + ("file", "C:\\other.db"), + ("file", "device_1/slice_00001.db"), + ("file", "device_0/slice_00000.db"), + ("file", "device_0/./slice_00001.db"), + ("file", "device_0/slice_00001.db\0"), + ], +) +def test_slice_overlap_gaps_capacity_paths_and_types(field, value): + data = manifest_data() + data["devices"]["0"]["slices"][1][field] = value + with pytest.raises(DatasetContractError, match=r"devices.0.slices\[1\]"): + parse_manifest(data) + + +def test_building_inspection_is_distinct_from_complete_analysis(): + data = manifest_data() + data["status"] = "building" + data["devices"]["0"]["readySlices"] = [1] + data["devices"]["0"]["slices"][0]["ready"] = False + inspection = parse_manifest(data, require_complete=False) + with pytest.raises(DatasetContractError, match="building"): + inspection.require_complete() + with pytest.raises(DatasetContractError, match="building"): + parse_manifest(data) + data["status"] = "complete" + with pytest.raises(DatasetContractError, match="not_ready"): + parse_manifest(data, require_complete=False) + + +def test_opaque_empty_hash_is_not_a_verified_content_digest(): + data = manifest_data() + data["cacheHash"] = "" + assert parse_manifest(data).cache_hash == "" + + +def test_multiple_devices_and_short_slices_are_legal(): + data = manifest_data() + data["devices"]["7"] = deepcopy(data["devices"]["0"]) + for index, item in enumerate(data["devices"]["7"]["slices"]): + item["file"] = f"device_7/slice_{index:05d}.db" + data["eventsPerSlice"] = 3 + assert len(parse_manifest(data).devices) == 2 + + +@pytest.mark.parametrize( + "sql,code", + [ + ("DELETE FROM trace_entry_0 WHERE id=2;", "event_identity"), + ("UPDATE trace_entry_0 SET id=4 WHERE id=3;", "event_identity"), + ("UPDATE trace_entry_0 SET action=8 WHERE id=2;", "event_action"), + ("UPDATE trace_entry_0 SET action=4 WHERE id=-1;", "event_action"), + ("UPDATE trace_entry_0 SET action=NULL WHERE id=2;", "event_action"), + ("UPDATE trace_entry_0 SET active='bad' WHERE id=2;", "event_values"), + ("UPDATE trace_entry_0 SET reserved=-1 WHERE id=2;", "event_values"), + ("UPDATE trace_entry_0 SET callstack=x'00' WHERE id=2;", "event_values"), + ("CREATE TABLE CALLSTACK (id INTEGER, callstack TEXT);", "layout"), + ("CREATE TABLE TRACE_ENTRY_1 (id INTEGER);", "device_tables"), + ( + "ALTER TABLE block_0 RENAME TO saved; " + "CREATE TABLE block_0 (id INTEGER, address INTEGER, size INTEGER, " + "requestedSize INTEGER, state INTEGER, allocEventId INTEGER, freeEventId INTEGER, " + "PRIMARY KEY (id,address)); INSERT INTO block_0 SELECT * FROM saved;", + "primary_key", + ), + ("UPDATE dictionary SET \"value\"='wrong' WHERE \"column\"='state';", "dictionary"), + ("CREATE TABLE trace_entry_1 (id INTEGER);", "device_tables"), + ("ALTER TABLE block_0 RENAME TO block_1;", "device_tables"), + ("CREATE TABLE callstack (id INTEGER, callstack TEXT);", "layout"), + ("ALTER TABLE trace_entry_0 RENAME COLUMN callstack TO callstackId;", "columns"), + ("ALTER TABLE trace_entry_0 ADD COLUMN extra INTEGER;", "columns"), + ( + "ALTER TABLE trace_entry_0 RENAME TO saved; CREATE VIEW trace_entry_0 AS SELECT * FROM saved;", + "table", + ), + ("UPDATE block_0 SET id=7 WHERE id=0;", "block_identity"), + ("UPDATE block_0 SET allocEventId=-2 WHERE id=-1;", "block_identity"), + ("UPDATE block_0 SET freeEventId=-2 WHERE id=0;", "lifecycle"), + ("INSERT INTO block_0 VALUES (2,300,8,8,1,2,1);", "lifecycle"), + ("UPDATE block_0 SET address=300 WHERE id=0;", "block_identity"), + ("UPDATE block_0 SET freeEventId=3 WHERE id=0;", "block_identity"), + ("UPDATE block_0 SET requestedSize=9 WHERE id=0;", "block"), + ("UPDATE block_0 SET state=99 WHERE id=0;", "block"), + ], +) +def test_invalid_sqlite_artifacts_are_locatable(tmp_path, sql, code): + root = make_dataset(tmp_path / "dataset") + mutate_db(root, sql) + with pytest.raises(DatasetContractError) as exc: + validate_dataset(root) + assert exc.value.code == code + assert exc.value.location.startswith("device_0/slice_00001.db") + + +def test_missing_slice_does_not_create_database(tmp_path): + root = make_dataset(tmp_path / "dataset") + path = root / "device_0/slice_00001.db" + path.unlink() + with pytest.raises(DatasetContractError, match="missing"): + validate_dataset(root) + assert not path.exists() + + +def test_symlink_artifact_member_rejected(tmp_path): + root = make_dataset(tmp_path / "dataset") + path = root / "device_0/slice_00001.db" + path.unlink() + path.symlink_to(root / "device_0/slice_00000.db") + with pytest.raises(DatasetContractError, match="symlink"): + validate_dataset(root) + + +def test_invalid_json_duplicate_members_and_sqlite_fail_closed(tmp_path): + root = make_dataset(tmp_path / "dataset") + path = root / "manifest.json" + original = path.read_text(encoding="utf-8") + for content in ( + "{", + original.replace('"schemaVersion": 1', '"schemaVersion": 1, "schemaVersion": 1'), + ): + path.write_text(content, encoding="utf-8") + with pytest.raises(DatasetContractError): + validate_dataset(root) + path.write_text(original, encoding="utf-8") + (root / "device_0/slice_00001.db").write_bytes(b"not a database") + with pytest.raises(DatasetContractError, match="sqlite"): + validate_dataset(root) + + +@pytest.mark.parametrize( + "scope", + [ + QueryScope("dataset"), + QueryScope("device", 0), + QueryScope("slice", 0, 1), + QueryScope("event_range", 0, None, 1, 3), + ], +) +def test_scope_contract_not_execution(scope): + scope.validate(parse_manifest(manifest_data())) + + +@pytest.mark.parametrize( + "scope", + [ + QueryScope("dataset", 0), + QueryScope("device", 1), + QueryScope("device", 0, 1), + QueryScope("slice", 0, 2), + QueryScope("event_range", 0, None, -1, 3), + QueryScope("event_range", 0, None, 3, 1), + QueryScope("event_range", 0, None, 0, 4), + ], +) +def test_invalid_scope(scope): + with pytest.raises(DatasetContractError): + scope.validate(parse_manifest(manifest_data())) + + +def test_pinned_msinsight_enum_facts(): + # Fixed-revision snapshot_db.py, not native v2's additional oom=8 enum. + assert ACTION_NAMES == ( + "segment_map", + "segment_unmap", + "segment_alloc", + "segment_free", + "alloc", + "free_requested", + "free_completed", + "workspace_snapshot", + ) + assert BLOCK_STATES == ((-1, "inactive"), (0, "active_pending_free"), (1, "active_allocated")) + + +def test_bilingual_manifest_examples_are_identical_and_valid(): + root = Path(__file__).resolve().parents[1] + examples = [] + for language in ("en", "zh"): + text = (root / f"docs/{language}/sharded-snapshotdb.md").read_text(encoding="utf-8") + example = text.split("```json\n", 1)[1].split("```", 1)[0] + data = json.loads(example) + assert parse_manifest(data).devices[0].event_count == 4 + examples.append(data) + assert examples[0] == examples[1] + + +def test_missing_manifest_and_symlink_directory_fail_closed(tmp_path): + with pytest.raises(DatasetContractError, match="missing"): + validate_dataset(tmp_path / "absent") + root = make_dataset(tmp_path / "dataset") + alias = tmp_path / "alias" + alias.symlink_to(root, target_is_directory=True) + with pytest.raises(DatasetContractError, match="path"): + validate_dataset(alias) + + +@pytest.mark.parametrize("value", [None, [], "manifest"]) +def test_invalid_manifest_object(value): + with pytest.raises(DatasetContractError, match="type"): + parse_manifest(value) diff --git a/tests/test_dataset_focus.py b/tests/test_dataset_focus.py new file mode 100644 index 0000000..8b525f4 --- /dev/null +++ b/tests/test_dataset_focus.py @@ -0,0 +1,504 @@ +"""Synthetic finalized artifacts: no pickle, metadata or optional reference tables.""" + +import hashlib +import json +import os +import sqlite3 +from contextlib import closing +from pathlib import Path + +import pytest +from typer.testing import CliRunner + +from pt_snap_cli.api import SnapshotAnalyzer +from pt_snap_cli.cli import app +from pt_snap_cli.completion import complete_device_ids +from pt_snap_cli.core.context_cache import ContextCache +from pt_snap_cli.core.dataset_contract import ( + ACTION_NAMES, + BLOCK_STATES, + DatasetContractError, + QueryScope, + validate_dataset, +) +from pt_snap_cli.core.dataset_resolver import DatasetResolver +from pt_snap_cli.core.errors import ( + DatabaseSchemaError, + InvalidDeviceError, + InvalidParameterError, + QueryExecutionError, + TemplateNotFoundError, + TemplateRenderError, +) +from pt_snap_cli.core.focus_service import FocusService + +runner = CliRunner() + + +@pytest.fixture(autouse=True) +def isolate(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + monkeypatch.setenv("HOME", str(tmp_path)) + monkeypatch.delenv("PT_SNAP_DB_PATH", raising=False) + monkeypatch.setattr(Path, "home", lambda: tmp_path) + + +def make_dataset(root, devices=(0, 2), slices=3): + root.mkdir() + data = { + "schemaVersion": 1, + "status": "complete", + "sourceFile": "/archived/missing.pkl", + "cacheHash": "", + "eventsPerSlice": 2, + "devices": {}, + } + for device in devices: + directory = root / f"device_{device}" + directory.mkdir() + records = [] + for index in range(slices): + file = f"device_{device}/slice_{index:05d}.db" + records.append( + { + "index": index, + "startEventId": index * 2, + "endEventId": index * 2 + 1, + "file": file, + "ready": True, + } + ) + with closing(sqlite3.connect(root / file)) as conn, conn: + conn.executescript( + 'CREATE TABLE dictionary ("table" TEXT,"column" TEXT,"key" TEXT,"value" TEXT);' + f"CREATE TABLE trace_entry_{device} (id INTEGER PRIMARY KEY, action INTEGER," + "address INTEGER,size INTEGER,stream INTEGER,allocated INTEGER,active INTEGER," + "reserved INTEGER,callstack TEXT);" + f"CREATE TABLE block_{device} (id INTEGER PRIMARY KEY,address INTEGER,size INTEGER," + "requestedSize INTEGER,state INTEGER,allocEventId INTEGER,freeEventId INTEGER);" + ) + conn.executemany( + "INSERT INTO dictionary VALUES (?,?,?,?)", + [ + (f"trace_entry_{device}", "action", str(key), value) + for key, value in enumerate(ACTION_NAMES) + ] + + [ + (f"block_{device}", "state", str(key), value) for key, value in BLOCK_STATES + ], + ) + conn.executemany( + f"INSERT INTO trace_entry_{device} VALUES (?,4,100,8,0,8,8,16,?)", + [ + (event, f"device{device}:event{event}") + for event in range(index * 2, index * 2 + 2) + ], + ) + conn.execute( + f"INSERT INTO trace_entry_{device} VALUES (-1,2,100,16,0,999,999,999,'boundary')" + ) + data["devices"][str(device)] = { + "eventCount": slices * 2, + "sliceCount": slices, + "readySlices": list(range(slices)), + "slices": records, + } + (root / "manifest.json").write_text(json.dumps(data), encoding="utf-8") + return root + + +def hashes(root): + return { + str(p.relative_to(root)): hashlib.sha256(p.read_bytes()).hexdigest() + for p in root.rglob("*") + if p.is_file() + } + + +@pytest.mark.parametrize("manifest_path", [False, True]) +def test_dataset_focus_readonly_and_complete_addressing(tmp_path, manifest_path): + root = make_dataset(tmp_path / "capture.msinsight") + path = root / "manifest.json" if manifest_path else root + before = hashes(root) + with SnapshotAnalyzer() as analyzer: + state = analyzer.set_focus(str(path), device_id=2) + assert state.available_devices == [0, 2] + assert state.callstack_layout == "v1" + assert not (tmp_path / ".pt-snap/focus.json").exists() + overview = analyzer.get_database_overview() + assert overview["devices"] == [ + {"device_id": d, "first_event_id": 0, "last_event_id": 5} for d in (0, 2) + ] + assert overview["dataset"]["format"] == "compatibility-v1" + assert overview["dataset"]["structured_frames"] is False + assert overview["dataset"]["text_callstacks"] is True + assert overview["dataset"]["cross_slice_queries"] is True + assert overview["dataset"]["boundary_event_count"] == 6 + assert analyzer.get_database_metadata()["status"] == "unavailable" + for device in (0, 2): + for event in range(6): + result = analyzer.execute_query("event", {"id": event}, device_id=device) + assert result["rows"][0]["callstack"] == f"device{device}:event{event}" + assert result["scope"]["slice_index"] == event // 2 + assert len(analyzer.context_cache) <= 4 + dataset = DatasetResolver().inspect(path) + assert dataset is not None + assert len(dataset.paths(QueryScope("dataset"))) == 6 + assert len(dataset.paths(QueryScope("device", 2))) == 3 + assert len(dataset.paths(QueryScope("event_range", 2, start_event_id=1, end_event_id=4))) == 3 + assert before == hashes(root) + assert not list(root.rglob("*-journal")) + assert not list(root.rglob("*-wal")) + + +def test_precedence_device_and_completion(tmp_path, monkeypatch): + root = make_dataset(tmp_path / "dataset") + service = FocusService() + service.set_global_focus(root, 0) + service.set_project_focus(root / "manifest.json", 2) + with SnapshotAnalyzer() as analyzer: + assert analyzer.execute_query("event", {"id": 4})["device_id"] == 2 + assert analyzer.execute_query("event", {"id": 4}, device_id=0)["device_id"] == 0 + assert complete_device_ids() == ["0", "2"] + monkeypatch.setenv("PT_SNAP_DB_PATH", str(root)) + with SnapshotAnalyzer() as analyzer: + assert analyzer.execute_query("event", {"id": 4})["device_id"] == 0 + with SnapshotAnalyzer(root / "manifest.json", 2) as analyzer: + assert analyzer.execute_query("event", {"id": 4})["device_id"] == 2 + before = json.loads((tmp_path / ".pt-snap/focus.json").read_text()) + monkeypatch.delenv("PT_SNAP_DB_PATH") + state = service.set_device(0) + assert state.available_devices == [0, 2] + assert set(json.loads((tmp_path / ".pt-snap/focus.json").read_text())) == { + "db_path", + "device_id", + } + assert set(before) == {"db_path", "device_id"} + + +@pytest.mark.parametrize( + "template,params,slice_index,error", + [ + ("event", {"id": -1}, None, InvalidParameterError), + ("event", {"id": 6}, None, InvalidParameterError), + ("event", {"id": 4}, 0, InvalidParameterError), + ("event", {"min_id": -1}, 0, InvalidParameterError), + ("event", {"min_id": 3, "max_id": 2}, None, InvalidParameterError), + ("event", {}, 99, InvalidParameterError), + ("event", {}, True, InvalidParameterError), + ("event", {"unknown": 0}, 0, TemplateRenderError), + ("leak_detection", {}, 0, QueryExecutionError), + ("missing_template", {}, None, TemplateNotFoundError), + ], +) +def test_unsupported_scope_errors(tmp_path, template, params, slice_index, error): + root = make_dataset(tmp_path / "dataset") + with SnapshotAnalyzer(root) as analyzer, pytest.raises(error): + analyzer.execute_query(template, params, slice_index=slice_index) + + +def test_slice_and_single_slice_range_exclude_boundary_and_preserve_filters(tmp_path): + root = make_dataset(tmp_path / "dataset") + with SnapshotAnalyzer(root) as analyzer: + result = analyzer.execute_query("event", slice_index=1, exact_total=True) + assert [row["id"] for row in result["rows"]] == [2, 3] + assert result["total"] == 2 and result["total_is_exact"] + assert result["scope"]["boundary_events_included"] is False + assert analyzer.execute_query("event", {"min_id": 2, "max_id": 3})["rows"] == result["rows"] + assert analyzer.execute_query("event", {"id": 2, "min_id": 3})["rows"] == [] + with pytest.raises(InvalidDeviceError): + analyzer.execute_query("event", {"id": 0}, device_id=99) + with pytest.raises(InvalidDeviceError): + analyzer.set_focus(str(root), device_id=99) + + +@pytest.mark.parametrize( + "failure", + ["building", "version", "missing", "duplicate", "symlink", "wal", "corrupt", "readiness"], +) +def test_invalid_dataset_is_not_partial_focus(tmp_path, failure): + root = make_dataset(tmp_path / "dataset") + manifest = root / "manifest.json" + data = json.loads(manifest.read_text()) + if failure in ("building", "version", "readiness"): + if failure == "building": + data["status"] = "building" + elif failure == "version": + data["schemaVersion"] = 999 + else: + data["devices"]["0"]["slices"][0]["ready"] = False + manifest.write_text(json.dumps(data)) + elif failure == "duplicate": + manifest.write_text( + manifest.read_text().replace( + '"schemaVersion": 1', '"schemaVersion": 1, "schemaVersion": 1' + ) + ) + elif failure == "symlink": + # Keep original data in place; replace the directory input with an alias. + alias = tmp_path / "alias" + alias.symlink_to(root, target_is_directory=True) + root = alias + elif failure == "wal": + (root / "device_0/slice_00000.db-wal").touch() + elif failure == "corrupt": + (root / "device_0/slice_00000.db").write_bytes(b"not SQLite") + else: + (root / "device_0/slice_00000.db").unlink() + with SnapshotAnalyzer() as analyzer: + with pytest.raises(ValueError): + analyzer.set_focus(str(root)) + assert analyzer.get_focus().db_path is None + with pytest.raises(ValueError): + analyzer.get_database_overview(str(root)) + with pytest.raises(DatabaseSchemaError): + FocusService().set_project_focus(root) + assert not (tmp_path / ".pt-snap/focus.json").exists() + + +def test_content_generation_reopens_cached_context_and_borrowed_close(tmp_path): + root = make_dataset(tmp_path / "dataset") + cache = ContextCache(maxsize=1) + try: + with SnapshotAnalyzer(root, context_cache=cache) as analyzer: + analyzer.execute_query("event", {"id": 0}) + path = root / "device_0/slice_00000.db" + old = cache.get(path, generation=DatasetResolver().inspect(root).fingerprint) + stamp = path.stat() + with closing(sqlite3.connect(path)) as conn, conn: + conn.execute("UPDATE trace_entry_0 SET callstack='modified:evt0' WHERE id=0") + os.utime(path, ns=(stamp.st_atime_ns, stamp.st_mtime_ns)) + assert ( + analyzer.execute_query("event", {"id": 0})["rows"][0]["callstack"] + == "modified:evt0" + ) + assert old._conn is None + manifest = root / "manifest.json" + data = json.loads(manifest.read_text()) + data["cacheHash"] = "new-generation" + manifest.write_text(json.dumps(data)) + analyzer.execute_query("event", {"id": 0}) + analyzer.execute_query("event", {"id": 4}) + assert len(cache) == 1 + assert len(cache) == 1 # Injected cache still caller-owned. + with pytest.raises(RuntimeError, match="closed"): + analyzer.get_focus() + finally: + cache.close() + assert len(cache) == 0 + + +def test_single_db_native_and_inline_keep_original_query_behavior(tmp_path): + root = make_dataset(tmp_path / "dataset") + path = root / "device_0/slice_00000.db" + for native in (False, True): + if native: + with closing(sqlite3.connect(path)) as conn, conn: + conn.execute("CREATE TABLE callstack (id INTEGER PRIMARY KEY, callstack TEXT)") + conn.execute("INSERT INTO callstack VALUES (0, 'native')") + conn.execute("ALTER TABLE trace_entry_0 DROP COLUMN callstack") + conn.execute("ALTER TABLE trace_entry_0 ADD COLUMN callstackId INTEGER") + conn.execute("UPDATE trace_entry_0 SET callstackId=0") + with SnapshotAnalyzer(path) as analyzer: + assert analyzer.get_focus().callstack_layout == ("v2" if native else "v1") + assert analyzer.execute_query("event", {"id": -1})["rows"][0]["id"] == -1 + assert analyzer.get_database_overview()["devices"][0]["first_event_id"] == 0 + with pytest.raises(InvalidParameterError): + analyzer.execute_query("event", slice_index=0) + if native: + with pytest.raises(DatabaseSchemaError): + DatasetResolver().inspect(root) # Never label native v2 as compatibility v1. + + +def test_no_manifest_private_staging_is_rejected(tmp_path): + root = tmp_path / "private-native-staging" + root.mkdir() + with pytest.raises(DatabaseSchemaError, match="manifest.json"): + DatasetResolver().inspect(root) + assert not list(root.iterdir()) + + +def test_single_slice_default_is_full_real_trace(tmp_path): + root = make_dataset(tmp_path / "dataset", slices=1) + with SnapshotAnalyzer(root) as analyzer: + assert [row["id"] for row in analyzer.execute_query("event")["rows"]] == [0, 1] + + +def test_cli_releases_owned_context_and_sqlite_is_readonly(tmp_path, monkeypatch): + root = make_dataset(tmp_path / "dataset") + opened = [] + original = ContextCache.get + + def get(cache, path, **kwargs): + context = original(cache, path, **kwargs) + opened.append(context) + with context.connect() as conn, pytest.raises(sqlite3.OperationalError, match="readonly"): + conn.execute("CREATE TABLE illegal (id INTEGER)") + return context + + monkeypatch.setattr(ContextCache, "get", get) + result = runner.invoke( + app, ["query", str(root), "--template-use", "event", "--params", '{"id": 0}', "--json"] + ) + assert result.exit_code == 0, result.output + assert opened and all(context._conn is None for context in opened) + failure = runner.invoke( + app, + [ + "query", + str(root), + "--template-use", + "event", + "--params", + '{"id": 0}', + "--timeout", + "not-a-number", + "--json", + ], + ) + assert failure.exit_code != 0 + assert all(context._conn is None for context in opened) + + +@pytest.mark.parametrize("suffix", ["-wal", "-journal", "-shm"]) +@pytest.mark.parametrize("dangling", [False, True]) +@pytest.mark.parametrize("bare", [False, True]) +def test_live_sidecar_rejected_before_any_sqlite_open( + tmp_path, monkeypatch, suffix, dangling, bare +): + root = make_dataset(tmp_path / "dataset") + # The last member must be checked before even the first slice is opened. + sidecar = root / f"device_2/slice_00002.db{suffix}" + if dangling: + sidecar.symlink_to(tmp_path / "missing-sidecar-target") + assert sidecar.is_symlink() and not sidecar.exists() + else: + sidecar.touch() + before = hashes(root) + inventory = sorted(str(p.relative_to(root)) for p in root.rglob("*")) + + def forbidden(*args, **kwargs): + raise AssertionError("SQLite opened before live sidecar rejection") + + monkeypatch.setattr(sqlite3, "connect", forbidden) + with pytest.raises( + DatasetContractError if bare else DatabaseSchemaError, match="live SQLite sidecar" + ): + validate_dataset(root) if bare else DatasetResolver().inspect(root) + assert before == hashes(root) + assert inventory == sorted(str(p.relative_to(root)) for p in root.rglob("*")) + if dangling: + assert sidecar.readlink() == tmp_path / "missing-sidecar-target" + + +@pytest.mark.parametrize("manifest_path", [False, True]) +@pytest.mark.parametrize("operation", ["inspect", "p0", "focus", "api", "cli"]) +def test_checkpointed_wal_read_without_artifact_changes( + tmp_path, monkeypatch, manifest_path, operation +): + root = make_dataset(tmp_path / "dataset") + member = root / "device_2/slice_00002.db" + # A genuine checkpointed WAL database retains header versions 2/2 after + # the last writer closes, even though no WAL/SHM file remains on disk. + with closing(sqlite3.connect(member)) as conn: + assert conn.execute("PRAGMA journal_mode=WAL").fetchone() == ("wal",) + with member.open("rb") as source: + assert source.read(20)[18:20] == b"\x02\x02" + assert not list(root.rglob("*-wal")) and not list(root.rglob("*-shm")) + before = hashes(root) + inventory = sorted(str(p.relative_to(root)) for p in root.rglob("*")) + opened = [] + original = sqlite3.connect + + def connect(*args, **kwargs): + opened.append(args[0]) + return original(*args, **kwargs) + + monkeypatch.setattr(sqlite3, "connect", connect) + path = root / "manifest.json" if manifest_path else root + if operation == "inspect": + assert DatasetResolver().inspect(path).validation.real_event_count == 12 + elif operation == "p0": + assert validate_dataset(root).real_event_count == 12 + elif operation == "focus": + assert FocusService().set_project_focus(path).callstack_layout == "v1" + elif operation == "api": + with SnapshotAnalyzer(path) as analyzer: + assert analyzer.execute_query("event", {"id": 5}, device_id=2)["rows"][0]["id"] == 5 + assert analyzer.execute_query("event", {"id": 5}, device_id=2)["rows"][0]["id"] == 5 + else: + result = runner.invoke( + app, + [ + "query", + str(path), + "--template-use", + "event", + "--device", + "2", + "--params", + '{"id":5}', + "--json", + ], + ) + assert result.exit_code == 0, result.output + assert json.loads(result.output)["rows"][0]["id"] == 5 + assert before == hashes(root) + assert inventory == sorted(str(p.relative_to(root)) for p in root.rglob("*")) + assert opened and all("mode=ro&immutable=1" in str(uri) for uri in opened) + assert not list(root.rglob("*-wal")) and not list(root.rglob("*-shm")) + + +@pytest.mark.parametrize("alias_parent", [False, True]) +@pytest.mark.parametrize("bare", [False, True]) +def test_member_alias_rejected_before_header_read(tmp_path, monkeypatch, alias_parent, bare): + source = make_dataset(tmp_path / "source", devices=(0,), slices=1) + root = tmp_path / "dataset" + root.mkdir() + (root / "manifest.json").write_bytes((source / "manifest.json").read_bytes()) + if alias_parent: + (root / "device_0").symlink_to(source / "device_0", target_is_directory=True) + else: + (root / "device_0").mkdir() + (root / "device_0/slice_00000.db").symlink_to(source / "device_0/slice_00000.db") + before = hashes(source) + original = Path.open + + def open_path(path, *args, **kwargs): + if path.suffix == ".db": + raise AssertionError("Member alias followed before header rejection") + return original(path, *args, **kwargs) + + monkeypatch.setattr(Path, "open", open_path) + with pytest.raises(DatasetContractError if bare else DatabaseSchemaError, match="symlink"): + validate_dataset(root) if bare else DatasetResolver().inspect(root) + monkeypatch.setattr(Path, "open", original) + assert before == hashes(source) + + +def test_manifest_change_during_validation_fails(tmp_path, monkeypatch): + import pt_snap_cli.core.dataset_resolver as module + + root = make_dataset(tmp_path / "dataset") + original = module.validate_dataset + + def validate(path): + result = original(path) + manifest = root / "manifest.json" + data = json.loads(manifest.read_text()) + data["cacheHash"] = "replaced" + manifest.write_text(json.dumps(data)) + return result + + monkeypatch.setattr(module, "validate_dataset", validate) + with pytest.raises(DatabaseSchemaError, match="changed during inspection"): + DatasetResolver().inspect(root) + + +def test_unknown_extension_never_promotes_frames(tmp_path): + root = make_dataset(tmp_path / "dataset") + path = root / "manifest.json" + data = json.loads(path.read_text()) + data["extensions"] = {"frames": {"version": 999}} + path.write_text(json.dumps(data)) + assert DatasetResolver().inspect(root).to_dict()["structured_frames"] is False diff --git a/tests/test_dataset_import_baseline.py b/tests/test_dataset_import_baseline.py new file mode 100644 index 0000000..b3d3533 --- /dev/null +++ b/tests/test_dataset_import_baseline.py @@ -0,0 +1,173 @@ +"""Structural benchmark contracts, never speed assertions or fixture execution.""" + +import hashlib +import json +import subprocess +import sys +from pathlib import Path + +import pytest + +from benchmarks.dataset_import_baseline import rss_kib, source_identity + + +@pytest.mark.parametrize( + "platform,raw,expected", [("darwin", 1048576, 1024), ("linux", 1024, 1024)] +) +def test_isolated_self_rss_units(platform, raw, expected): + assert rss_kib(raw, platform) == expected + + +def test_source_gate_checks_exact_hash_and_size_without_pickle_load(tmp_path): + source = tmp_path / "not-an-executable-fixture.txt" + source.write_bytes(b"hash only") + digest = hashlib.sha256(source.read_bytes()).hexdigest() + assert source_identity(source, digest)["bytes"] == 9 + with pytest.raises(ValueError, match="exact reviewed"): + source_identity(source, "0" * 64) + link = tmp_path / "link" + link.symlink_to(source) + with pytest.raises(ValueError, match="canonical"): + source_identity(link, digest) + + +def test_existing_parent_refused_before_import_and_preserved(tmp_path): + source = tmp_path / "hash-only.txt" + source.write_bytes(b"not pickle; must not load") + script = Path(__file__).resolve().parents[1] / "benchmarks/dataset_import_baseline.py" + result = subprocess.run( + [ + sys.executable, + str(script), + "--source", + str(source), + "--sha256", + hashlib.sha256(source.read_bytes()).hexdigest(), + "--device", + "0", + "--capacity", + "2", + "--output", + str(tmp_path), + "--trusted-pickle", + ], + capture_output=True, + text=True, + ) + assert result.returncode != 0 and "FileExistsError" in result.stderr + assert source.read_bytes() == b"not pickle; must not load" + assert sorted(p.name for p in tmp_path.iterdir()) == [source.name] + + +def test_oversized_source_rejected_before_open(tmp_path, monkeypatch): + source = tmp_path / "large.pkl" + source.write_bytes(b"x" * (2 * 1024 * 1024 + 1)) + monkeypatch.setattr(Path, "open", lambda *_a, **_k: pytest.fail("oversized source opened")) + with pytest.raises(ValueError, match="small source"): + source_identity(source, "0" * 64) + + +@pytest.mark.parametrize( + "flags", [["--mode", "single-db"], ["--mode", "single-db", "--phase", "cold"]] +) +def test_hidden_flags_require_pair_and_owner_before_pickle(tmp_path, flags): + source = tmp_path / "inert.txt" + source.write_bytes(b"not pickle") + script = Path(__file__).resolve().parents[1] / "benchmarks/dataset_import_baseline.py" + result = subprocess.run( + [ + sys.executable, + str(script), + "--source", + str(source), + "--sha256", + hashlib.sha256(source.read_bytes()).hexdigest(), + "--device", + "0", + "--capacity", + "2", + "--output", + str(tmp_path / "missing"), + "--trusted-pickle", + *flags, + ], + capture_output=True, + text=True, + ) + assert result.returncode == 2 and "supplied together" in result.stderr + assert not (tmp_path / "missing").exists() + + +def test_reuse_miss_refuses_rebuild_and_preserves_existing_actual_bytes(tmp_path): + from benchmarks.dataset_import_baseline import cache_hashes + + source = tmp_path / "inert.pkl" + source.write_bytes(b"not pickle") + output = tmp_path / "single-db" + output.mkdir() + old = output / "inert.pkl.db" + old.write_bytes(b"old invalid destination must not be replaced") + before = old.read_bytes() + digest = hashlib.sha256(source.read_bytes()).hexdigest() + envelope = tmp_path / "run-envelope.json" + envelope.write_text( + json.dumps( + { + "source": source_identity(source, digest), + "device": 0, + "capacity": 2, + "cacheHashes": {"single-db": cache_hashes(output)}, + } + ) + ) + script = Path(__file__).resolve().parents[1] / "benchmarks/dataset_import_baseline.py" + result = subprocess.run( + [ + sys.executable, + str(script), + "--source", + str(source), + "--sha256", + digest, + "--device", + "0", + "--capacity", + "2", + "--output", + str(output), + "--trusted-pickle", + "--mode", + "single-db", + "--phase", + "reuse", + "--envelope", + str(envelope), + ], + capture_output=True, + text=True, + ) + assert result.returncode != 0 and "refuse deserialization" in result.stderr + assert old.read_bytes() == before and sorted(p.name for p in output.iterdir()) == [old.name] + + +def test_recorded_performance_report_is_structural_not_a_speed_gate(): + from pt_snap_cli.core.dataset_support import DATASET_SUPPORT + + repository = Path(__file__).resolve().parents[1] + raw = (repository / "docs/dataset-performance-baseline.json").read_text() + report = json.loads(raw) + assert "/Users/" not in raw and "/private/" not in raw + assert report["schema_version"] == 1 and report["GUI"] == "pending/not-run" + assert len(report["source_head"]) == 40 and len(report["benchmark_script_sha256"]) == 64 + assert {(r["mode"], r["phase"]) for r in report["records"]} == { + (mode, phase) + for mode in ("single-db", "pt-snap-native-v2", "compatibility-v1") + for phase in ("cold", "reuse") + } + for record in report["records"]: + assert record["exit"] == 0 and record["phase_rss"]["status"] == "unavailable" + assert "RUSAGE_SELF" in record["rss_scope"] + assert set(record["queries"]) == set(DATASET_SUPPORT) + assert record["individual_shard_bytes"] and record["total_disk_bytes"] > 0 + assert all(q["status"] == "passed" for q in record["queries"].values()) + # Deliberately no timing/RSS/speedup threshold: these are observations. diff --git a/tests/test_dataset_import_cli.py b/tests/test_dataset_import_cli.py new file mode 100644 index 0000000..704cdfd --- /dev/null +++ b/tests/test_dataset_import_cli.py @@ -0,0 +1,81 @@ +import json +from pathlib import Path + +import pytest +from typer.testing import CliRunner + +from pt_snap_cli.api import SnapshotAnalyzer +from pt_snap_cli.cli import _safe_call, app +from tests.core.test_dataset_import import source_options + + +@pytest.fixture(autouse=True) +def isolate(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + monkeypatch.setenv("HOME", str(tmp_path)) + monkeypatch.setattr(Path, "home", lambda: tmp_path) + monkeypatch.delenv("PT_SNAP_DB_PATH", raising=False) + + +def test_dataset_cli_json_generation_reuse_force_focus_and_text(tmp_path): + options = source_options(tmp_path, multiple=True) + args = [ + "import", + str(options.snapshot_file), + "-o", + str(options.output_dir), + "--events-per-slice", + "2", + ] + runner = CliRunner() + result = runner.invoke(app, [*args, "--json"]) + assert result.exit_code == 0, result.output + data = json.loads(result.output) + assert data["ok"] is True + assert data["format"] == "pt-snap-native-v2" + assert data["devices"] == [0, 1] and data["slice_count"] == 10 + assert data["omitted_devices"] == [2] + assert data["dataset_path"] == data["db_path"] + assert data["rebuilt"] and not data["reused"] + assert data["focus_state"]["callstack_layout"] == "v2" + reused = runner.invoke(app, [*args, "--no-focus", "--json"]) + assert reused.exit_code == 0 + assert json.loads(reused.output)["reused"] is True + rebuilt = runner.invoke(app, [*args, "--force"]) + assert rebuilt.exit_code == 0 and "slices: 10" in rebuilt.output + focus = runner.invoke(app, ["focus", data["dataset_path"], "--device", "1", "--json"]) + assert focus.exit_code == 0 + focused = json.loads(focus.output) + with SnapshotAnalyzer(data["dataset_path"], device_id=1) as analyzer: + state = analyzer.get_focus() + assert state.db_path == focused["db_path"] + assert state.available_devices == focused["available_devices"] == [0, 1] + assert state.callstack_layout == focused["callstack_layout"] == "v2" + assert state.device_id == focused["device_id"] == 1 + + +@pytest.mark.parametrize( + "arguments", + [ + ["--events-per-slice", "0"], + ["--events-per-slice", "nope"], + ["--events-per-slice"], + ["--format", "unknown"], + ["--format", ""], + ["--format", "pt-snap-native-v2"], + ["--events-per-slice", "2", "--format", "single-db"], + ["--events-per-slice", "2", "--format", "compatibility-v2"], + ["--device", "-1"], + ], +) +def test_dataset_cli_argument_error_json(tmp_path, arguments, monkeypatch, capsys): + options = source_options(tmp_path) + monkeypatch.setattr( + "sys.argv", ["pt-snap", "import", str(options.snapshot_file), "--json", *arguments] + ) + assert _safe_call() != 0 + captured = capsys.readouterr() + assert captured.out == "" + data = json.loads(captured.err) + assert data["ok"] is False and data["error"]["code"] == "INVALID_PARAMETER" + assert not list(tmp_path.glob("*.pt-snap-native-v2")) diff --git a/tests/test_fragmentation_query_scope.py b/tests/test_fragmentation_query_scope.py index 6fb6b61..2cab6ce 100644 --- a/tests/test_fragmentation_query_scope.py +++ b/tests/test_fragmentation_query_scope.py @@ -126,8 +126,13 @@ def test_explicit_full_trace_comparison_uses_its_own_peak(scope_db): assert sum(row["size_bytes"] for row in report["callstack_groups"]) == peak["peak_active"] -def test_full_trace_synthetic_peak_requires_stopping_comparison(scope_db): +def test_full_trace_real_peak_excludes_synthetic_boundary_and_drives_report(scope_db): section = SKILL_PATH.read_text().split("#### Separate full-trace comparison", 1)[1] - peak = run_skill_command(skill_commands(section)[0], scope_db)["rows"][0] - assert peak["peak_active_event_id"] == -2 - assert peak["peak_active"] == 1000 + commands = skill_commands(section.split("### 6.", 1)[0]) + peak = run_skill_command(commands[0], scope_db)["rows"][0] + assert peak["peak_active_event_id"] == 25 + assert peak["peak_active"] == 90 # never the synthetic id=-2/value=1000 + report = run_skill_command(commands[1], scope_db) + assert report["event_id"] == 25 + assert report["active_bytes_at_event"] == 90 + assert report["included_bytes"] == 90 and report["coverage_percent"] == 100 diff --git a/tests/test_interop_acceptance.py b/tests/test_interop_acceptance.py new file mode 100644 index 0000000..ef87be7 --- /dev/null +++ b/tests/test_interop_acceptance.py @@ -0,0 +1,319 @@ +"""Offline acceptance: generated SQLite is synthetic, NEVER original-producer/GUI evidence.""" + +import dataclasses +import json +import os +import sqlite3 +import subprocess +import sys +from contextlib import closing +from pathlib import Path + +import pytest +from typer.testing import CliRunner + +from benchmarks.interop_acceptance import ( + REVISION, + cli_acceptance, + compare_artifact, + compare_forward, + inventory, + physical_oracle, + read_json, + verify_receipt, +) +from pt_snap_cli.api import SnapshotAnalyzer +from pt_snap_cli.cli import app +from pt_snap_cli.core.dataset_support import DATASET_SUPPORT +from pt_snap_cli.core.errors import ImportExecutionError +from pt_snap_cli.core.import_service import ImportService +from tests.core.test_dataset_attribution import add_frames, case +from tests.core.test_msinsight_export import options + + +@pytest.fixture(autouse=True) +def isolate(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + monkeypatch.setenv("HOME", str(tmp_path)) + monkeypatch.setattr(Path, "home", lambda: tmp_path) + monkeypatch.delenv("PT_SNAP_DB_PATH", raising=False) + + +@pytest.mark.parametrize("device", [0, 2]) +def test_all_eleven_contracts_against_independent_physical_reference(tmp_path, device): + root = case(tmp_path) + result = compare_artifact(root, device, tmp_path / "reference.db") + assert set(result["templates"]) == set(DATASET_SUPPORT) + assert result["real_events"] == 16 and result["canonical_lifecycles"] == 10 + assert result["raw_block_rows"] == 80 and result["repeated_block_rows"] == 70 + assert result["GUI"] == "human-deferred/pending/not-run" + assert result["original_producer"] == "not-run by this runner" + + +def test_hash_first_receipt_rejects_changed_members_and_dangling_sidecars(tmp_path): + root = case(tmp_path) + receipt = {"revision": REVISION, "artifactHashes": inventory(root)} + verify_receipt(root, receipt) + manifest = root / "manifest.json" + manifest.write_bytes(manifest.read_bytes() + b" ") + with pytest.raises(ValueError, match="checksum"): + verify_receipt(root, receipt) + receipt["artifactHashes"] = inventory(root) + (root / "device_0/slice_00000.db-wal").symlink_to(tmp_path / "missing") + with pytest.raises(ValueError, match="symlink"): + verify_receipt(root, receipt) + + +def test_reference_is_exclusive_and_ambiguous_address_is_not_identity(tmp_path): + root = case(tmp_path) + reference = tmp_path / "reference.db" + reference.write_bytes(b"do not replace") + with pytest.raises(FileExistsError): + compare_artifact(root, 0, reference) + assert reference.read_bytes() == b"do not replace" + oracle = physical_oracle(root, 0) + assert len([b for b in oracle["blocks"].values() if b["address"] == 100]) == 2 + with closing(sqlite3.connect(root / "device_0/slice_00007.db")) as conn, conn: + conn.execute("UPDATE block_0 SET size=size+1 WHERE id=0") + with pytest.raises(ValueError, match="Ambiguous"): + physical_oracle(root, 0) + + +def test_whole_dataset_focus_cli_api_report_and_truncated_scope(tmp_path): + root = case(tmp_path) + before = inventory(root) + runner = CliRunner() + focus = runner.invoke(app, ["focus", str(root / "manifest.json"), "--device", "0", "--json"]) + assert focus.exit_code == 0, focus.output + query_args = [ + "query", + "--template-use", + "event", + "--params", + '{"order_by":"active","order_dir":"DESC","limit":3,"offset":1}', + "-n", + "2", + "--exact-total", + "--json", + ] + cli = runner.invoke(app, query_args) + assert cli.exit_code == 0, cli.output + result = json.loads(cli.output) + with SnapshotAnalyzer(root) as analyzer: + api = analyzer.execute_query( + "event", + {"order_by": "active", "order_dir": "DESC", "limit": 3, "offset": 1}, + device_id=0, + max_rows=2, + exact_total=True, + ) + for key in ("rows", "total", "has_more", "truncated", "total_is_exact", "scope"): + assert result[key] == api[key] + assert result["scope"]["range_complete"] and result["truncated"] + assert analyzer.get_database_overview()["import_metadata"]["reason"] == "metadata_missing" + report = runner.invoke( + app, + [ + "report", + "peak-memory", + str(root), + "--device", + "0", + "--metric", + "reserved", + "--start-id", + "4", + "--end-id", + "15", + "--limit", + "-1", + "--json", + ], + ) + assert report.exit_code == 0, report.output + data = json.loads(report.output) + assert data["source_coverage"]["allocation_source_complete"] + assert data["scope"]["range_complete"] + assert inventory(root) == before + + +@pytest.mark.parametrize("version,corrupt", [(1, None), (999, None), (1, "gap"), (1, "absent")]) +def test_reader_only_frames_evidence_is_distinct_from_original_text(tmp_path, version, corrupt): + root = case(tmp_path) + frame = add_frames(root, version, corrupt) + with SnapshotAnalyzer(root) as analyzer: + row = analyzer.execute_query("event", {"id": 0, "stack_bytes": 0})["rows"][0] + if version == 1 and corrupt is None: + assert row["frames"] == [frame, frame, {"name": "event:0"}] + else: + assert row["frames"] is None and row["frames_status"] == "text_only" + + +def test_compatible_owned_cache_reuse_and_external_force_preserve_actual_bytes(tmp_path): + request = dataclasses.replace(options(tmp_path, focus=False), set_focus=False) + service = ImportService() + result = service.import_snapshot(request) + before = inventory(result.db_path) + assert service.import_snapshot(request).reused + assert service.import_snapshot(dataclasses.replace(request, force=True)).reused + manifest = result.db_path / "manifest.json" + raw = json.loads(manifest.read_text()) + raw.pop("ptSnap") # external original-shaped cache, not our ownership attestation + manifest.write_text(json.dumps(raw)) + external = inventory(result.db_path) + with pytest.raises(ImportExecutionError, match="preserv"): + service.import_snapshot(dataclasses.replace(request, force=True)) + assert inventory(result.db_path) == external + assert before != external + + +@pytest.mark.parametrize("kind", ["trace", "block", "dictionary"]) +def test_physical_row_bounds_include_negative_trace_and_dictionary(tmp_path, kind): + root = case(tmp_path) + with closing(sqlite3.connect(root / "device_0/slice_00000.db")) as conn, conn: + if kind == "trace": + conn.executemany( + "INSERT INTO trace_entry_0 VALUES (?,2,0,0,0,0,0,0,'')", + ((-100 - n,) for n in range(20001)), + ) + elif kind == "block": + conn.executemany( + "INSERT INTO block_0 VALUES (?,0,0,0,1,-1,-1)", ((-100 - n,) for n in range(20001)) + ) + else: + conn.executemany( + "INSERT INTO dictionary VALUES ('extra','extra',?,'x')", + ((str(n),) for n in range(257)), + ) + with pytest.raises(ValueError, match="20000|256 rows"): + physical_oracle(root, 0) + + +def test_regular_bounded_json_source_and_inventory_before_materialization(tmp_path, monkeypatch): + large = tmp_path / "large.json" + large.write_bytes(b" " * (1024 * 1024 + 1)) + with pytest.raises(ValueError, match="1 MiB"): + read_json(large) + with pytest.raises(ValueError, match="regular"): + read_json(tmp_path) + large.write_text('{"a":1,"a":2}') + with pytest.raises(ValueError, match="Duplicate"): + read_json(large) + root = case(tmp_path) + receipt = {"revision": REVISION, "artifactHashes": inventory(root)} + source = tmp_path / "too-large.pkl" + source.write_bytes(b"x" * (2 * 1024 * 1024 + 1)) + with pytest.raises(ValueError, match="small hash-only"): + verify_receipt(root, receipt, source) + for n in range(257): + (root / f"extra-{n}").touch() + with pytest.raises(ValueError, match="256-entry"): + inventory(root) + + +def test_acceptance_output_cannot_mutate_an_input_artifact(tmp_path): + root = case(tmp_path) + before = inventory(root) + with pytest.raises(ValueError, match="overlap"): + compare_artifact(root, 0, root / "new-reference.db") + assert inventory(root) == before + + +def test_forward_physical_contract_detects_changed_observation(tmp_path): + first = tmp_path / "first" + second = tmp_path / "second" + first.mkdir() + second.mkdir() + original = case(first) + compatible = case(second) + assert compare_forward(original, compatible, 0)["status"] == "passed" + with closing(sqlite3.connect(compatible / "device_0/slice_00007.db")) as conn, conn: + conn.execute("UPDATE block_0 SET state=0 WHERE id=3") + with pytest.raises(AssertionError, match="forward physical"): + compare_forward(original, compatible, 0) + + +@pytest.mark.parametrize( + "change", + [ + "stable_tokens", + "size", + "state", + "multiplicity", + "real_zero_id", + "real_size", + "unstable_token", + ], +) +def test_forward_token_namespaces_preserve_all_fields_and_multiplicity(tmp_path, change): + first, second = tmp_path / "first", tmp_path / "second" + first.mkdir() + second.mkdir() + original, compatible = case(first), case(second) + for index in range(8): + with closing(sqlite3.connect(compatible / f"device_0/slice_{index:05d}.db")) as conn, conn: + if change == "stable_tokens": + conn.execute("UPDATE block_0 SET id=id-1000 WHERE id<0") + elif change == "size": + conn.execute("UPDATE block_0 SET size=size+1 WHERE id=-21") + elif change == "state": + conn.execute("UPDATE block_0 SET state=0 WHERE id=-21") + elif change == "multiplicity": + conn.execute( + "INSERT INTO block_0 SELECT -999,address,size,requestedSize,state,allocEventId,freeEventId FROM block_0 WHERE id=-21" + ) + elif change == "real_zero_id": + conn.execute("UPDATE block_0 SET id=999 WHERE id=0") + elif change == "real_size": + conn.execute("UPDATE block_0 SET size=size+1 WHERE id=0") + elif index == 7: + conn.execute("UPDATE block_0 SET address=address+1 WHERE id=-21") + if change == "stable_tokens": + assert "UNPROVED" in compare_forward(original, compatible, 0)["negative_identity"] + else: + with pytest.raises((AssertionError, ValueError), match="forward physical|Ambiguous"): + compare_forward(original, compatible, 0) + + +def test_real_console_module_chain_writes_only_owned_test_focus(tmp_path): + root = case(tmp_path) + before = inventory(root) + output = tmp_path / "evidence" + output.mkdir() + records = cli_acceptance(root, 0, output) + assert len(records) == 3 and all(r["exit"] == 0 for r in records) + assert (output / ".pt-snap/focus.json").is_file() + assert inventory(root) == before + + +def test_cli_subprocess_relative_pythonpath_keeps_exact_worktree_source(tmp_path): + root = case(tmp_path) + receipt = tmp_path / "receipt.json" + receipt.write_text(json.dumps({"revision": REVISION, "artifactHashes": inventory(root)})) + repository = Path(__file__).resolve().parents[1] + output = tmp_path / "evidence" + result = subprocess.run( + [ + sys.executable, + "benchmarks/interop_acceptance.py", + "--artifact", + str(root), + "--receipt", + str(receipt), + "--device", + "0", + "--output", + str(output), + ], + cwd=repository, + env={**os.environ, "PYTHONPATH": "src"}, + capture_output=True, + text=True, + ) + assert result.returncode == 0, result.stderr + records = json.loads((output / "acceptance.json").read_text())["cli_acceptance"] + assert len(records) == 3 + assert all(r["source_api"] == str(repository / "src/pt_snap_cli/api.py") for r in records) + assert all( + r["env"]["PYTHONPATH"].split(os.pathsep)[0] == str(repository / "src") for r in records + ) diff --git a/tests/test_native_dataset.py b/tests/test_native_dataset.py new file mode 100644 index 0000000..07672f6 --- /dev/null +++ b/tests/test_native_dataset.py @@ -0,0 +1,215 @@ +import dataclasses +import json +import sqlite3 +from contextlib import closing +from pathlib import Path + +import pytest + +from pt_snap_cli.api import SnapshotAnalyzer +from pt_snap_cli.core.context_cache import ContextCache +from pt_snap_cli.core.dataset_contract import DatasetContractError, QueryScope, parse_manifest +from pt_snap_cli.core.dataset_resolver import DatasetResolver, _hash_file +from pt_snap_cli.core.errors import DatabaseSchemaError, InvalidParameterError +from pt_snap_cli.core.import_service import ImportService +from pt_snap_cli.core.native_dataset_contract import parse_native_manifest +from tests.core.test_dataset_import import hashes, source_options + + +@pytest.fixture(autouse=True) +def isolate(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + monkeypatch.setenv("HOME", str(tmp_path)) + monkeypatch.setattr(Path, "home", lambda: tmp_path) + monkeypatch.delenv("PT_SNAP_DB_PATH", raising=False) + + +def dataset(tmp_path): + return ( + ImportService() + .import_snapshot( + dataclasses.replace(source_options(tmp_path, multiple=True), set_focus=False) + ) + .db_path + ) + + +def test_native_sparse_ids_positions_device_scope_borrowed_cache_and_readonly(tmp_path): + root = dataset(tmp_path) + before = hashes(root) + cache = ContextCache(maxsize=2) + try: + with SnapshotAnalyzer(root, context_cache=cache) as analyzer: + assert analyzer.get_database_metadata()["status"] == "available" + overview = analyzer.get_database_overview() + assert overview["devices"] == [ + {"device_id": d, "first_event_id": 7, "last_event_id": 87} for d in (0, 1) + ] + assert overview["dataset"]["real_event_count"] == 18 + assert overview["dataset"]["structured_frames"] is False + for device in (0, 1): + for position in range(9): + result = analyzer.execute_query( + "event", {"id": position * 10 + 7}, device_id=device + ) + assert result["rows"][0]["id"] == position * 10 + 7 + assert result["scope"]["slice_index"] == position // 2 + assert len(cache) <= 2 + assert len(analyzer.execute_query("event", {}, device_id=1, slice_index=0)["rows"]) == 2 + peak = analyzer.execute_query("memory_peak") + assert peak["scope"]["slice_indices"] == [0, 1, 2, 3, 4] + assert peak["scope"]["boundary_events_included"] is False + assert peak["rows"][0]["peak_active_event_id"] in range(7, 88, 10) + assert len(cache) <= 2 + with pytest.raises(InvalidParameterError): + analyzer.execute_query("event", {"id": -1}) + held = cache.get(root / "device_1/slice_00000.db") + assert len(cache) > 0 + with held.connect() as conn: + assert conn.execute("SELECT 1").fetchone()[0] == 1 + resolved = DatasetResolver().inspect(root / "manifest.json") + assert len(resolved.paths(QueryScope("dataset"))) == 10 + assert ( + len(resolved.paths(QueryScope("event_range", 0, start_event_id=7, end_event_id=87))) + == 5 + ) + with pytest.raises(InvalidParameterError): + resolved.paths(QueryScope("event_range", 0, start_event_id=0, end_event_id=87)) + finally: + cache.close() + assert hashes(root) == before + assert not list(root.rglob("*-wal")) and not list(root.rglob("*-journal")) + raw = json.loads((root / "manifest.json").read_text()) + with pytest.raises(DatasetContractError): + parse_manifest(raw) # Never relabel native-v2 as compatibility-v1. + + +@pytest.mark.parametrize( + "part", + ["manifest", "device", "member", "wal", "journal", "shm", "dangling-wal", "persistent-wal"], +) +def test_native_alias_sidecar_and_persistent_wal_checks_precede_any_sqlite_open( + tmp_path, monkeypatch, part +): + root = dataset(tmp_path) + member = root / "device_1/slice_00004.db" + if part in ("manifest", "device", "member"): + path = {"manifest": root / "manifest.json", "device": root / "device_1", "member": member}[ + part + ] + original = tmp_path / ("original-" + part) + path.rename(original) + path.symlink_to(original, target_is_directory=part == "device") + elif part == "persistent-wal": + with closing(sqlite3.connect(member)) as conn: + assert conn.execute("PRAGMA journal_mode=WAL").fetchone() == ("wal",) + assert not Path(str(member) + "-wal").exists() + else: + sidecar = Path(str(member) + "-" + part.removeprefix("dangling-")) + if part.startswith("dangling"): + sidecar.symlink_to(tmp_path / "missing-sidecar") + else: + sidecar.write_bytes(b"live") + monkeypatch.setattr( + sqlite3, "connect", lambda *a, **k: pytest.fail("unsafe artifact opened SQLite") + ) + with pytest.raises(DatabaseSchemaError): + DatasetResolver().inspect(root) + + +@pytest.mark.parametrize( + "field,value", + [ + ("format", "compatibility-v1"), + ("schemaVersion", 99), + ("status", "building"), + ("omittedDevices", [0]), + ("omittedDevices", [2, 2]), + ], +) +def test_native_manifest_publication_and_version_validation(tmp_path, field, value): + root = dataset(tmp_path) + raw = json.loads((root / "manifest.json").read_text()) + raw[field] = value + with pytest.raises(DatasetContractError): + parse_native_manifest(raw) + + +@pytest.mark.parametrize( + "field,value", + [("eventCount", 89), ("sliceCount", True), ("readySlices", [0]), ("sliceCount", 3)], +) +def test_native_manifest_count_and_ready_validation(tmp_path, field, value): + root = dataset(tmp_path) + raw = json.loads((root / "manifest.json").read_text()) + raw["devices"]["0"][field] = value + with pytest.raises(DatasetContractError): + parse_native_manifest(raw) + + +@pytest.mark.parametrize( + "field,value", + [ + ("startPosition", 1), + ("endPosition", 999), + ("index", True), + ("startEventId", -1), + ("endEventId", 0), + ("file", "../escape.db"), + ("sha256", "bad"), + ("ready", False), + ], +) +def test_native_slice_validation(tmp_path, field, value): + root = dataset(tmp_path) + raw = json.loads((root / "manifest.json").read_text()) + raw["devices"]["0"]["slices"][0][field] = value + with pytest.raises(DatasetContractError): + parse_native_manifest(raw) + + +@pytest.mark.parametrize( + "sql", + [ + "UPDATE trace_entry_0 SET action=99 WHERE id=7", + "UPDATE block_0 SET id=id+1000 WHERE allocEventId>=0", + "DELETE FROM pt_snap_block_reference", + "UPDATE pt_snap_metadata SET source_name='wrong'", + "DELETE FROM dictionary", + "DROP TABLE callstack", + "UPDATE trace_entry_0 SET callstackId=9999 WHERE id=7", + ], +) +def test_full_native_validation_not_only_manifest_or_member_hash(tmp_path, sql): + root = dataset(tmp_path) + member = root / "device_0/slice_00000.db" + with closing(sqlite3.connect(member)) as conn, conn: + conn.execute(sql) + raw = json.loads((root / "manifest.json").read_text()) + raw["devices"]["0"]["slices"][0]["sha256"] = _hash_file(member) + (root / "manifest.json").write_text(json.dumps(raw)) + with pytest.raises(DatabaseSchemaError): + DatasetResolver().inspect(root) + + +def test_unknown_frame_extension_never_promotes_structured_capability_and_generation_changes( + tmp_path, +): + root = dataset(tmp_path) + cache = ContextCache() + try: + with SnapshotAnalyzer(root, context_cache=cache) as analyzer: + analyzer.execute_query("event", {"id": 7}) + old = next(iter(cache._entries.values()))[0] + old_connection = old._conn + raw = json.loads((root / "manifest.json").read_text()) + raw["extensions"] = {"structuredFrames": {"version": 999, "present": True}} + (root / "manifest.json").write_text(json.dumps(raw)) + analyzer.execute_query("event", {"id": 7}) + assert next(iter(cache._entries.values()))[0] is not old + assert analyzer.get_database_overview()["dataset"]["structured_frames"] is False + assert not hasattr(analyzer, "import_snapshot") + with pytest.raises(sqlite3.ProgrammingError): + old_connection.execute("SELECT 1") + finally: + cache.close()