diff --git a/README.md b/README.md index fda331b..8ac0dc0 100644 --- a/README.md +++ b/README.md @@ -52,6 +52,10 @@ deterministic names, replay validation, and failure-safe publication. See the [full quick start guide](docs/en/quickstart.md) for a walkthrough. +Complete compatibility-v1 sharded artifacts can be focused by directory or +`manifest.json`, without the archived pickle. See [dataset focus and addressing](docs/en/sharded-snapshotdb.md#complete-dataset-focus-and-addressing-p1) +for read-only overview, event routing, `--slice`, and explicit cross-slice limits. + ## Commands | Command | Description | diff --git a/README_zh.md b/README_zh.md index 31e5cec..7c814e4 100644 --- a/README_zh.md +++ b/README_zh.md @@ -47,6 +47,10 @@ pt-snap split snapshot.pkl --slices 4 --output snapshot-slices 完整的入门指南见 [Quick Start](docs/zh/quickstart.md)。 +完整兼容 v1 分片产物可以目录或 `manifest.json` 为 focus,无需已归档 pickle。 +只读 overview、事件路由、`--slice` 和跨片限制见 +[完整数据集 focus 与寻址](docs/zh/sharded-snapshotdb.md#完整数据集-focus-与寻址p1)。 + ## 命令 | 命令 | 说明 | diff --git a/docs/en/sharded-snapshotdb.md b/docs/en/sharded-snapshotdb.md index a5e396a..65a9d5b 100644 --- a/docs/en/sharded-snapshotdb.md +++ b/docs/en/sharded-snapshotdb.md @@ -7,8 +7,9 @@ This freezes an artifact contract and shared validation models, **not a new CLI exporter or cross-slice query engine**. Existing `import` still defaults to one native v2 DB; `split` still produces replayable pickle/JSON. Existing single-DB -v1/v2 analysis and focus are unchanged. The `complete` gate below applies only to -this new dataset contract, not to legacy single-DB analysis. +v1/v2 single-DB analysis is unchanged. The P1 reader below additionally accepts a +complete compatibility-v1 manifest or directory as focus. The `complete` gate +applies only to datasets, not to legacy single-DB analysis. `pt_snap_cli.core.dataset_contract` provides `parse_manifest`, `validate_event_ids`, `validate_dataset`, immutable manifest/device/slice models, @@ -166,9 +167,9 @@ which P0 does not implement. Text stacks cannot substitute for these guarantees. | `event_range` | `device_id`, inclusive `start_event_id/end_event_id` within the real trace, possibly across slices | There is no global event order across devices. Boundary IDs cannot be range -endpoints. `query_execution=False` explicitly declares that cross-slice execution -is deferred; passing scope validation does not make existing CLI/API accept a -manifest or directory. +endpoints. P0 `DatasetValidation.query_execution=False` describes validation, +not execution. P1 accepts complete manifests/directories through the resolver +below; cross-slice execution remains deferred. ```python from pt_snap_cli.core.dataset_contract import QueryScope, validate_dataset @@ -180,6 +181,75 @@ QueryScope("event_range", device_id=0, start_event_id=1, end_event_id=3).validat assert not inspection.structured_frames ``` +## Complete dataset focus and addressing (P1) + +A relocated compatibility-v1 directory or its `manifest.json` can be focused +without the original pickle, import metadata, or reference/extension tables: + +```bash +pt-snap focus /capture/snapshot.pkl.msinsight --device 0 --json +pt-snap overview --json +pt-snap query --template-use event --params '{"id": 3}' --json +pt-snap query --template-use event --slice 1 --json +``` + +`overview.dataset` reports format, manifest version/status, content fingerprint, +real/boundary counts, per-device slice paths/ranges and capabilities. Event bounds +exclude negative synthetic rows (also for standalone DB overview). Text stacks +are available, but not necessarily populated; structured frames and cross-slice +queries are **unavailable**, including when unknown extensions claim them. +Dataset-wide import metadata is unavailable; per-slice metadata is not a source +attestation for the entire dataset. + +Path precedence stays explicit → `PT_SNAP_DB_PATH` → nearest project focus → legacy +global config; device precedence stays explicit → focused → first discovered. +CLI focus only writes the chosen project/global focus file, never the artifact or +detected layout. API session focus does not write it: + +```python +from pathlib import Path +from pt_snap_cli import SnapshotAnalyzer + +with SnapshotAnalyzer(Path("/capture/snapshot.pkl.msinsight"), device_id=0) as analyzer: + event = analyzer.execute_query("event", {"id": 3}) + page = analyzer.execute_query("event", slice_index=1, exact_total=True) + print(event["scope"], page["total_is_exact"]) +``` + +Only the `event` template is currently supported through dataset focus: a real ID +routes to its containing slice; an inclusive `min_id/max_id` range must fit one +slice; `--slice`/`slice_index` lists that slice's real events. Negative IDs are not +whole-dataset selectors. Output `scope` identifies the actual DB/device/slice, +restricted real interval, dataset fingerprint and boundary exclusion. Totals and +pagination apply to this scope, **not** to an implicitly truncated full dataset. +Caller parameter defaults in CLI `effective_params` are additionally constrained +by `scope`. Cross-slice ranges, unbounded multi-slice queries and aggregate or +lifecycle templates fail explicitly; they never silently choose the first/latest +slice. Direct single-DB v1/v2 queries retain their existing semantics. + +`core.dataset_resolver.DatasetResolver.inspect(path)` returns an immutable +`ResolvedDataset` (or `None` for a standalone DB). Its `paths(QueryScope(...))` +addresses all matching files even for a cross-slice range, but does not execute it. +It revalidates the finalized P0 contract on each call and hashes the manifest and +members; no unbounded manifest/connection cache is retained. Hashing uses bounded +chunks and is proportional to artifact size, not a performance claim. Context LRU +size defaults to four; dataset generation changes invalidate contexts on next +lookup even when size/mtime are unchanged. Analyzer-owned caches close on exit; +injected caches remain caller-owned. Validation opens/closes slices sequentially. + +Building, missing/inconsistent slices, unknown base versions and unsafe paths +are rejected before focus writes or query execution. Before opening **any** slice, +the resolver checks every member's canonical non-symlink path, SQLite header and +WAL/journal/SHM sidecars (including dangling symlinks). Persistent WAL mode is +rejected even after checkpointing has removed its sidecars: `mode=ro` can recreate +them. The reader does not repair, checkpoint, delete sidecars or change journal +mode; standalone DB behavior is unchanged. This is read-only inspection, not a +filesystem sandbox or a concurrent producer lock. Native-v2 replay's private staging directory has **no published manifest** +and is rejected as a dataset; its individual DBs remain readable via the existing +single-DB entry. A future compatibility-v1 producer can use this same reader +without producer-specific metadata. No native-v2 manifest protocol, exporter, +cache publication, migration, or cross-slice lifecycle reconstruction is invented. + ## Internal continuous replay (P1) `core.sharded_replay_service.ShardedReplayService.stage(source, directory, diff --git a/docs/zh/sharded-snapshotdb.md b/docs/zh/sharded-snapshotdb.md index bd687b7..9c0159c 100644 --- a/docs/zh/sharded-snapshotdb.md +++ b/docs/zh/sharded-snapshotdb.md @@ -6,7 +6,8 @@ 本阶段冻结产物协议和共享校验模型,**不新增 CLI 导出器或跨片查询引擎**。现有 `import` 默认仍生成一个原生 v2 DB;`split` 仍输出可回放 pickle/JSON。现有单库 -v1/v2 分析与 focus 不变。下述 `complete` 门槛只约束新数据集合同,不约束旧单库分析。 +v1/v2 单库分析不变。下述 P1 reader 新增将 complete 兼容 v1 manifest 或目录作为 +focus;`complete` 门槛只约束数据集,不约束旧单库分析。 `pt_snap_cli.core.dataset_contract` 提供 `parse_manifest`、`validate_event_ids`、 `validate_dataset`、不可变 manifest/device/slice 模型、`QueryScope` 和带 @@ -135,8 +136,9 @@ dictionary 的十进制整数 key 以文本存储,使用真实带设备后缀 | `slice` | `device_id`, `slice_index`;该片真实区间,边界独立 | | `event_range` | `device_id`、设备内闭区间 `start_event_id/end_event_id`;可跨片 | -设备间没有全局事件顺序。边界 ID 不能作为范围端点。`query_execution=False` -明确表示跨片执行留后续;scope 校验成功不使现有 CLI/API 自动接受 manifest 或目录。 +设备间没有全局事件顺序。边界 ID 不能作为范围端点。P0 的 +`DatasetValidation.query_execution=False` 描述校验而非执行。下述 P1 resolver +使 CLI/API 接受完整 manifest/目录;跨片执行仍留后续。 ```python from pt_snap_cli.core.dataset_contract import QueryScope, validate_dataset @@ -148,6 +150,60 @@ QueryScope("event_range", device_id=0, start_event_id=1, end_event_id=3).validat assert not inspection.structured_frames ``` +## 完整数据集 focus 与寻址(P1) + +已搬迁的兼容 v1 目录或其 `manifest.json` 可直接 focus;不要求原始 pickle、导入 +metadata、辅助引用表或扩展表: + +```bash +pt-snap focus /capture/snapshot.pkl.msinsight --device 0 --json +pt-snap overview --json +pt-snap query --template-use event --params '{"id": 3}' --json +pt-snap query --template-use event --slice 1 --json +``` + +`overview.dataset` 输出格式、manifest 版本/状态、内容指纹、真实/边界事件数、逐设备 +分片路径/范围及能力。事件范围排除负 ID 合成行(单库 overview 也如此)。支持文本调用栈, +不保证每行有栈;结构化 frames 与跨片查询**不可用**,未知扩展的能力声明不会提升它们。 +数据集级导入 metadata 为 unavailable;逐片 metadata 不是整份数据集的来源证明。 + +路径优先级仍为显式 → `PT_SNAP_DB_PATH` → 最近项目 focus → legacy global config; +设备优先级仍为显式 → focused → 首个发现设备。CLI focus 只写选定的项目/全局 focus 文件, +不写产物或检测布局;API 会话 focus 不写配置: + +```python +from pathlib import Path +from pt_snap_cli import SnapshotAnalyzer + +with SnapshotAnalyzer(Path("/capture/snapshot.pkl.msinsight"), device_id=0) as analyzer: + event = analyzer.execute_query("event", {"id": 3}) + page = analyzer.execute_query("event", slice_index=1, exact_total=True) + print(event["scope"], page["total_is_exact"]) +``` + +数据集 focus 当前只支持 `event` 模板:真实 ID 路由至所在片,闭区间 `min_id/max_id` +必须落在一个片内,`--slice`/`slice_index` 列出该片真实事件。负 ID 不能作为整份数据集 +selector。输出 `scope` 标识实际 DB/device/slice、受限真实区间、数据集指纹和边界排除。 +总数和分页针对该 scope,**不是**隐式截断的整份数据集。CLI `effective_params` 的调用者 +参数默认值另受 `scope` 约束。跨片区间、无界多片查询、聚合/生命周期模板明确失败,不会 +静默选首片/最新片。直接对单库 v1/v2 查询保持原有语义。 + +`core.dataset_resolver.DatasetResolver.inspect(path)` 返回不可变 `ResolvedDataset` +(单库返回 `None`);其 `paths(QueryScope(...))` 可寻址跨片范围的全部文件,但不执行查询。 +每次调用重新校验最终化 P0 合同并对 manifest/所有成员取内容哈希,不保存无界 manifest/ +连接缓存。哈希按有界块读取,工作量与产物大小成正比,不宣称性能收益。Context LRU 默认 +最多四个;数据集 generation 变化在下次 lookup 使旧 Context 失效,即使 size/mtime 相同。 +analyzer 自有缓存在退出时关闭,注入缓存仍由调用者所有;校验逐片打开并关闭连接。 + +building、缺片/矛盾片、未知基础版本、不安全路径在 focus 写入或查询前拒绝。在打开 +**任何**分片前,resolver 先检查全部成员的规范非符号链接路径、SQLite header 及 +WAL/journal/SHM sidecar(包含悬空符号链接)。即使 checkpoint 已移除 sidecar,持久 WAL +模式仍被拒绝:`mode=ro` 也可能重新创建它们。reader 不修复、不执行 checkpoint、不删除 +sidecar、不改变 journal mode;单库行为不变。这是只读检查,不是文件系统沙箱或并发生产端锁。native-v2 replay 的私有 +staging **没有已发布 manifest**,不能作为数据集;其中单个 DB 仍由已有单库入口读取。 +后续兼容 v1 生产端可直接使用同一 reader,不依赖生产端专属 metadata。本项不虚构 +native-v2 manifest 协议,不新增导出、缓存发布、迁移或跨片生命周期重建。 + ## 内部持续回放(P1) `core.sharded_replay_service.ShardedReplayService.stage(source, directory, diff --git a/src/pt_snap_cli/api.py b/src/pt_snap_cli/api.py index 6ca2d90..1e462ef 100644 --- a/src/pt_snap_cli/api.py +++ b/src/pt_snap_cli/api.py @@ -32,7 +32,11 @@ class TemplateSummaryPayload(TypedDict): category: str | None -class QueryResultPayload(TypedDict): +class _QueryScopePayload(TypedDict, total=False): + scope: dict[str, object] + + +class QueryResultPayload(_QueryScopePayload): total: int returned: int device_id: int | None @@ -209,6 +213,7 @@ def execute_query( *, exact_total: bool = False, timeout_s: float | None = None, + slice_index: int | None = None, ) -> QueryResultPayload: self._ensure_open() try: @@ -220,10 +225,11 @@ def execute_query( max_rows=max_rows, exact_total=exact_total, timeout_s=timeout_s, + slice_index=slice_index, ) except FocusNotConfiguredError as exc: raise RuntimeError("No database configured. Call set_focus() first.") from exc - return { + payload: QueryResultPayload = { "total": result.total, "returned": result.returned, "device_id": result.device_id, @@ -235,6 +241,9 @@ def execute_query( "total_is_exact": result.total_is_exact, "timeout_s": result.timeout_s, } + if result.scope is not None: + payload["scope"] = result.scope + return payload def list_capabilities(self) -> dict[str, object]: self._ensure_open() diff --git a/src/pt_snap_cli/cli.py b/src/pt_snap_cli/cli.py index 620f864..fcb2a27 100644 --- a/src/pt_snap_cli/cli.py +++ b/src/pt_snap_cli/cli.py @@ -94,6 +94,7 @@ json_success, ) from pt_snap_cli.core.error_codes import DATABASE_NOT_FOUND, ERROR, INVALID_PARAMETER +from pt_snap_cli.core.errors import InvalidParameterError from pt_snap_cli.query.config import OUTPUT_COLUMN_OPTIONAL from pt_snap_cli.query.executor import reported_sql_limit from pt_snap_cli.query.registry import discover_categories, get_query @@ -236,7 +237,12 @@ def main( @app.command("focus") def focus_database( - db_path: Annotated[Path | None, typer.Argument(help="Path to SQLite database file")] = None, + db_path: Annotated[ + Path | None, + typer.Argument( + help="Path to SQLite database, complete manifest.json, or dataset directory" + ), + ] = None, device: Annotated[ int | None, typer.Option( @@ -257,7 +263,7 @@ def focus_database( if db_path is None and device is None: try: state = focus_service.get_focus() - except FocusFileInvalidError as e: + except (FocusFileInvalidError, DatabaseSchemaError) as e: _error_from_exc(e) if json_output: @@ -297,6 +303,7 @@ def focus_database( FocusFileInvalidError, FocusNotConfiguredError, DatabaseMissingError, + DatabaseSchemaError, InvalidDeviceError, ) as e: _error_from_exc(e) @@ -454,7 +461,10 @@ def split_snapshot( @app.command("metadata") def show_database_metadata( db_path: Annotated[ - Path | None, typer.Argument(help="Path to database file (optional if configured)") + Path | None, + typer.Argument( + help="Path to database, manifest.json, or dataset directory (optional if configured)" + ), ] = None, json_output: Annotated[bool, _json_flag()] = False, ) -> None: @@ -491,7 +501,10 @@ def show_database_metadata( @app.command("query") def query_database( db_path: Annotated[ - Path | None, typer.Argument(help="Path to database file (optional if configured)") + Path | None, + typer.Argument( + help="Path to database, manifest.json, or dataset directory (optional if configured)" + ), ] = None, template_use: Annotated[ str | None, @@ -509,6 +522,10 @@ def query_database( autocompletion=complete_device_ids, ), ] = None, + slice_index: Annotated[ + int | None, + typer.Option("--slice", help="Explicit dataset slice index for event inspection"), + ] = None, list_templates: Annotated[ bool, typer.Option("--list", help="List all available query templates") ] = False, @@ -702,6 +719,7 @@ def query_database( max_rows=max_rows, exact_total=exact_total, timeout_s=timeout, + slice_index=slice_index, ) if json_output: resolved = focus_service.resolve_focus( @@ -725,6 +743,7 @@ def query_database( total_is_exact=result.total_is_exact, timeout_s=result.timeout_s, rows=result.rows, + **({"scope": result.scope} if result.scope is not None else {}), ) ) return @@ -779,6 +798,7 @@ def query_database( except ( TemplateRenderError, QueryExecutionError, + InvalidParameterError, DatabaseSchemaError, json.JSONDecodeError, ) as e: @@ -860,7 +880,10 @@ def show_capabilities( @app.command("overview") def show_database_overview( db_path: Annotated[ - Path | None, typer.Argument(help="Path to database file (optional if configured)") + Path | None, + typer.Argument( + help="Path to database, manifest.json, or dataset directory (optional if configured)" + ), ] = None, json_output: Annotated[bool, _json_flag()] = False, ) -> None: @@ -882,6 +905,12 @@ def show_database_overview( _emit_json(json_success(**service.overview_to_dict(overview))) return _print_database_overview(overview) + if overview.dataset is not None: + typer.echo(f"Dataset: {overview.dataset['format']} ({overview.dataset['status']})") + typer.echo(f"Slices/devices: {overview.dataset['devices']}") + typer.echo( + "Text callstacks available; structured frames and cross-slice queries unavailable." + ) # Flag-only options used when a parse failure happens before the --json callback. diff --git a/src/pt_snap_cli/completion.py b/src/pt_snap_cli/completion.py index abbd9cc..797dfff 100644 --- a/src/pt_snap_cli/completion.py +++ b/src/pt_snap_cli/completion.py @@ -75,6 +75,11 @@ def complete_device_ids() -> list[str]: return [] try: + from pt_snap_cli.core.dataset_resolver import DatasetResolver + + dataset = DatasetResolver().inspect(db_path) + if dataset is not None: + return [str(device) for device in dataset.device_ids] uri = f"file:{db_path}?mode=ro" with closing(sqlite3.connect(uri, uri=True)) as conn: cursor = conn.cursor() diff --git a/src/pt_snap_cli/context.py b/src/pt_snap_cli/context.py index c64b772..d11cb0a 100644 --- a/src/pt_snap_cli/context.py +++ b/src/pt_snap_cli/context.py @@ -204,7 +204,8 @@ def discover_devices(self) -> list[int]: def device_trace_bounds(self) -> list[tuple[int, int | None, int | None]]: """Return ``(device_id, first_event_id, last_event_id)`` for each device. - Bounds come from ``MIN(id)`` / ``MAX(id)`` on ``trace_entry_``. + Bounds come from nonnegative real IDs on ``trace_entry_``. + Negative synthetic boundary rows do not extend the real trace. Empty tables report ``None`` for both ends. The connection stays read-only (``mode=ro``). """ @@ -213,7 +214,7 @@ def device_trace_bounds(self) -> list[tuple[int, int | None, int | None]]: cursor = conn.cursor() for device_id in self.device_ids: table = f"trace_entry_{device_id}" - cursor.execute(f"SELECT MIN(id), MAX(id) FROM {_quote_ident(table)}") + cursor.execute(f"SELECT MIN(id), MAX(id) FROM {_quote_ident(table)} WHERE id >= 0") row = cursor.fetchone() first = int(row[0]) if row is not None and row[0] is not None else None last = int(row[1]) if row is not None and row[1] is not None else None diff --git a/src/pt_snap_cli/core/context_cache.py b/src/pt_snap_cli/core/context_cache.py index 296c6e0..cf22e0d 100644 --- a/src/pt_snap_cli/core/context_cache.py +++ b/src/pt_snap_cli/core/context_cache.py @@ -39,7 +39,9 @@ def __init__(self, maxsize: int = 4) -> None: raise ValueError("maxsize must be positive") self._maxsize = maxsize # Insertion order doubles as LRU recency (oldest first). - self._entries: OrderedDict[Path, tuple[Context, FileSignature]] = OrderedDict() + self._entries: OrderedDict[Path, tuple[Context, tuple[FileSignature, str | None]]] = ( + OrderedDict() + ) @property def maxsize(self) -> int: @@ -48,7 +50,7 @@ def maxsize(self) -> int: def __len__(self) -> int: return len(self._entries) - def get(self, db_path: Path | str) -> Context: + def get(self, db_path: Path | str, *, generation: str | None = None) -> Context: """Return a cached or freshly opened :class:`Context` for ``db_path``. On a hit the cached entry is promoted to most-recently-used. On a @@ -65,7 +67,9 @@ def get(self, db_path: Path | str) -> Context: # Probe metadata first so a missing/inaccessible file raises # consistently whether the cache is hit or missed. - current_signature = self._file_signature(key) + # A dataset reader can supply its validated manifest/member content identity. + # Single-DB callers retain the original filesystem-signature behavior. + current_signature = (self._file_signature(key), generation) cached = self._entries.get(key) if cached is not None: diff --git a/src/pt_snap_cli/core/dataset_resolver.py b/src/pt_snap_cli/core/dataset_resolver.py new file mode 100644 index 0000000..d7b5296 --- /dev/null +++ b/src/pt_snap_cli/core/dataset_resolver.py @@ -0,0 +1,169 @@ +"""Resolve finalized compatibility datasets above the single-DB Context layer. + +Original local reader of the P0 contract; no pickle, migration or producer code. +Native-v2 private replay staging has no published manifest and is not a dataset. +""" + +from __future__ import annotations + +import hashlib +import json +from dataclasses import dataclass +from pathlib import Path + +from pt_snap_cli.core.dataset_contract import ( + DatasetContractError, + DatasetValidation, + DeviceRecord, + QueryScope, + parse_manifest, + validate_dataset, +) +from pt_snap_cli.core.errors import DatabaseSchemaError, InvalidDeviceError, InvalidParameterError + + +@dataclass(frozen=True) +class ResolvedDataset: + root: Path + validation: DatasetValidation + fingerprint: str + + @property + def device_ids(self) -> list[int]: + return sorted(item.device_id for item in self.validation.manifest.devices) + + @property + def callstack_layout(self) -> str: + return "v1" + + @property + def callstack_layout_error(self) -> None: + return None + + def device(self, device_id: int | None = None) -> DeviceRecord: + selected = self.device_ids[0] if device_id is None else device_id + for device in self.validation.manifest.devices: + if type(selected) is int and device.device_id == selected: + return device + raise InvalidDeviceError(f"Device {selected} not found. Available: {self.device_ids}") + + def paths(self, scope: QueryScope) -> tuple[Path, ...]: + """Address all matching slices; this does not execute/aggregate queries.""" + try: + scope.validate(self.validation.manifest) + except DatasetContractError as exc: + raise InvalidParameterError(str(exc)) from exc + return tuple( + self.root / item.file + for device in self.validation.manifest.devices + if scope.device_id is None or device.device_id == scope.device_id + for item in device.slices + if (scope.slice_index is None or item.index == scope.slice_index) + and (scope.start_event_id is None or item.end_event_id >= scope.start_event_id) + and (scope.end_event_id is None or item.start_event_id <= scope.end_event_id) + ) + + def to_dict(self) -> dict[str, object]: + manifest = self.validation.manifest + return { + "manifest_path": str(self.root / "manifest.json"), + "schema_version": manifest.schema_version, + "status": manifest.status, + "format": "compatibility-v1", + "fingerprint": self.fingerprint, + "real_event_count": self.validation.real_event_count, + "boundary_event_count": self.validation.boundary_event_count, + "text_callstacks": True, + "structured_frames": False, + "cross_slice_queries": False, + "supported_queries": ["event: real ID, single-slice range, or explicit slice"], + "devices": [ + { + "device_id": device.device_id, + "event_count": device.event_count, + "slices": [ + { + "index": item.index, + "first_event_id": item.start_event_id, + "last_event_id": item.end_event_id, + "db_path": str(self.root / item.file), + } + for item in device.slices + ], + } + for device in sorted(manifest.devices, key=lambda item: item.device_id) + ], + } + + +class DatasetResolver: + """Stateless resolution: revalidate every call, retain no connections/manifest cache. + + Content hashes (not cacheHash or size/mtime alone) form the Context generation. + Work is proportional to artifact size; no concurrent-publication lock is claimed. + """ + + def inspect(self, path: Path | str) -> ResolvedDataset | None: + candidate = Path(path).expanduser().absolute() + if not candidate.is_dir() and candidate.name != "manifest.json": + return None + root = candidate if candidate.is_dir() else candidate.parent + if candidate.is_symlink() or root.resolve() != root: + raise DatabaseSchemaError("Dataset must use canonical non-symlink paths.") + try: + manifest_path = root / "manifest.json" + if manifest_path.is_symlink(): + raise DatabaseSchemaError("Dataset manifest must not be a symlink.") + before = _hash_file(manifest_path) + with manifest_path.open(encoding="utf-8") as source: + raw: object = json.load(source) + planned = parse_manifest(raw) + # mode=ro can create WAL/SHM files even after the last writer has + # checkpointed and removed them. Check every member before the first + # SQLite open, without following aliases or repairing the artifact. + for device in planned.devices: + for item in device.slices: + _require_readonly_member(root, root / item.file) + validation = validate_dataset(root) + digest = hashlib.sha256() + digest.update(before.encode("ascii")) + for device in validation.manifest.devices: + for item in device.slices: + member = root / item.file + _require_readonly_member(root, member) + digest.update(item.file.encode("utf-8")) + digest.update(_hash_file(member).encode("ascii")) + if before != _hash_file(manifest_path): + raise DatabaseSchemaError("Dataset manifest changed during inspection; retry.") + return ResolvedDataset(root, validation, digest.hexdigest()) + except (DatasetContractError, OSError, UnicodeError, json.JSONDecodeError) as exc: + raise DatabaseSchemaError(f"Invalid dataset: {exc}") from exc + + +def _require_readonly_member(root: Path, member: Path) -> None: + # Manifest parsing fixes canonical relative paths; reject member/parent + # symlinks before the header read, not only when the validator opens SQLite. + parts = member.relative_to(root).parts + for index in range(1, len(parts) + 1): + if root.joinpath(*parts[:index]).is_symlink(): + raise DatabaseSchemaError(f"Dataset member must not use symlink paths: {member}") + if member.resolve() != member or not member.is_file(): + raise DatabaseSchemaError(f"Dataset member is missing or not a canonical file: {member}") + for suffix in ("-wal", "-journal", "-shm"): + sidecar = Path(str(member) + suffix) + if sidecar.exists() or sidecar.is_symlink(): + raise DatabaseSchemaError(f"Dataset member has a live SQLite sidecar: {member}") + with member.open("rb") as source: + header = source.read(20) + # SQLite's file-format write/read versions at offsets 18/19 mark WAL as 2. + # A sidecar-free WAL database is still unsafe to open with mode=ro. + if header[:16] == b"SQLite format 3\x00" and 2 in header[18:20]: + raise DatabaseSchemaError(f"Dataset member uses persistent WAL mode: {member}") + + +def _hash_file(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as source: + for chunk in iter(lambda: source.read(1024 * 1024), b""): + digest.update(chunk) + return digest.hexdigest() diff --git a/src/pt_snap_cli/core/focus_service.py b/src/pt_snap_cli/core/focus_service.py index 45e31df..9e6015a 100644 --- a/src/pt_snap_cli/core/focus_service.py +++ b/src/pt_snap_cli/core/focus_service.py @@ -6,6 +6,7 @@ from pt_snap_cli.config import Config, FocusResolutionError from pt_snap_cli.context import Context, DatabaseNotFoundError, SchemaVersionError +from pt_snap_cli.core.dataset_resolver import DatasetResolver, ResolvedDataset from pt_snap_cli.core.errors import ( DatabaseMissingError, DatabaseSchemaError, @@ -55,7 +56,8 @@ def get_focus( if resolved.db_path is not None and resolved.db_path.exists(): try: - ctx = Context(resolved.db_path) + dataset = DatasetResolver().inspect(resolved.db_path) + ctx = dataset if dataset is not None else Context(resolved.db_path) available_devices = ctx.device_ids callstack_layout = ctx.callstack_layout callstack_layout_error = ctx.callstack_layout_error @@ -78,8 +80,9 @@ def set_project_focus( device_id: int | None = None, base_dir: Path | None = None, ) -> FocusState: - db_path = Path(db_path).expanduser().resolve() + db_path = Path(db_path).expanduser().absolute() ctx = self._validated_context(db_path) + db_path = db_path.resolve() self._validate_device(ctx, device_id) self._config.write_project_focus(db_path, base_dir=base_dir, device_id=device_id) return FocusState( @@ -93,8 +96,9 @@ def set_project_focus( ) def set_global_focus(self, db_path: Path | str, device_id: int | None = None) -> FocusState: - db_path = Path(db_path).expanduser().resolve() + db_path = Path(db_path).expanduser().absolute() ctx = self._validated_context(db_path) + db_path = db_path.resolve() self._validate_device(ctx, device_id) self._config.write_global_focus(db_path, device_id=device_id) return FocusState( @@ -161,8 +165,9 @@ def validate_session_db( db_path: Path | str, device_id: int | None = None, ) -> FocusState: - db_path = Path(db_path).expanduser().resolve() + db_path = Path(db_path).expanduser().absolute() ctx = self._validated_context(db_path) + db_path = db_path.resolve() self._validate_device(ctx, device_id) return FocusState( db_path=db_path, @@ -174,7 +179,10 @@ def validate_session_db( callstack_layout_error=ctx.callstack_layout_error, ) - def _validated_context(self, db_path: Path) -> Context: + def _validated_context(self, db_path: Path) -> Context | ResolvedDataset: + dataset = DatasetResolver().inspect(db_path) + if dataset is not None: + return dataset try: return Context(db_path) except DatabaseNotFoundError as exc: @@ -182,7 +190,7 @@ def _validated_context(self, db_path: Path) -> Context: except (SchemaVersionError, sqlite3.DatabaseError) as exc: raise DatabaseSchemaError(str(exc)) from exc - def _validate_device(self, ctx: Context, device_id: int | None) -> None: + def _validate_device(self, ctx: Context | ResolvedDataset, device_id: int | None) -> None: if device_id is None: return if device_id not in ctx.device_ids: diff --git a/src/pt_snap_cli/core/import_metadata.py b/src/pt_snap_cli/core/import_metadata.py index b2f9922..12c8747 100644 --- a/src/pt_snap_cli/core/import_metadata.py +++ b/src/pt_snap_cli/core/import_metadata.py @@ -11,6 +11,7 @@ from pt_snap_cli import __version__ from pt_snap_cli.context import Context, DatabaseNotFoundError, SchemaVersionError +from pt_snap_cli.core.dataset_resolver import DatasetResolver from pt_snap_cli.core.errors import DatabaseMissingError, DatabaseSchemaError, ImportMetadataError from pt_snap_cli.core.models import ( CacheDecision, @@ -76,7 +77,11 @@ def build_metadata( ) def inspect(self, db_path: Path | str) -> MetadataInspection: + dataset = DatasetResolver().inspect(db_path) path = Path(db_path).expanduser().resolve() + if dataset is not None: + # Per-slice extensions are not a dataset-wide importer/source attestation. + return MetadataInspection(path, "unavailable", reason="metadata_missing") try: context = Context(path) except DatabaseNotFoundError as exc: diff --git a/src/pt_snap_cli/core/models.py b/src/pt_snap_cli/core/models.py index 0201e26..b0ba150 100644 --- a/src/pt_snap_cli/core/models.py +++ b/src/pt_snap_cli/core/models.py @@ -107,6 +107,7 @@ class QueryResult: truncated: bool = False total_is_exact: bool = False timeout_s: float | None = None + scope: dict[str, object] | None = None @dataclass(frozen=True) @@ -264,6 +265,7 @@ class DatabaseOverview: focus_source: FocusSource devices: list[DeviceTraceBounds] metadata: MetadataInspection + dataset: dict[str, object] | None = None @dataclass(frozen=True) diff --git a/src/pt_snap_cli/core/overview_service.py b/src/pt_snap_cli/core/overview_service.py index ea974ef..a109c88 100644 --- a/src/pt_snap_cli/core/overview_service.py +++ b/src/pt_snap_cli/core/overview_service.py @@ -6,6 +6,7 @@ from pt_snap_cli.context import Context, DatabaseNotFoundError, SchemaVersionError from pt_snap_cli.core.context_cache import ContextCache +from pt_snap_cli.core.dataset_resolver import DatasetResolver from pt_snap_cli.core.errors import ( DatabaseMissingError, DatabaseSchemaError, @@ -13,7 +14,7 @@ ) from pt_snap_cli.core.focus_service import FocusService from pt_snap_cli.core.import_metadata import ImportMetadataService -from pt_snap_cli.core.models import DatabaseOverview, DeviceTraceBounds +from pt_snap_cli.core.models import DatabaseOverview, DeviceTraceBounds, MetadataInspection class OverviewService: @@ -48,7 +49,21 @@ def inspect( if resolved.db_path is None: raise FocusNotConfiguredError("No database path specified and no database configured.") + dataset = DatasetResolver().inspect(resolved.db_path) db_path = resolved.db_path.expanduser().resolve() + if dataset is not None: + return DatabaseOverview( + db_path=db_path, + focus_source=resolved.source, + devices=[ + DeviceTraceBounds(device.device_id, 0, device.event_count - 1) + for device in sorted( + dataset.validation.manifest.devices, key=lambda d: d.device_id + ) + ], + metadata=MetadataInspection(db_path, "unavailable", reason="metadata_missing"), + dataset=dataset.to_dict(), + ) ctx = self._validated_context(db_path) try: raw_bounds = ctx.device_trace_bounds() @@ -74,6 +89,7 @@ def overview_to_dict(self, overview: DatabaseOverview) -> dict[str, Any]: return { "db_path": str(overview.db_path), "focus_source": overview.focus_source, + **({"dataset": overview.dataset} if overview.dataset is not None else {}), "devices": [ { "device_id": device.device_id, diff --git a/src/pt_snap_cli/core/query_service.py b/src/pt_snap_cli/core/query_service.py index d0ff782..d5013f3 100644 --- a/src/pt_snap_cli/core/query_service.py +++ b/src/pt_snap_cli/core/query_service.py @@ -9,6 +9,8 @@ from pt_snap_cli.context import Context, DatabaseNotFoundError, SchemaVersionError from pt_snap_cli.core.context_cache import ContextCache +from pt_snap_cli.core.dataset_contract import QueryScope +from pt_snap_cli.core.dataset_resolver import DatasetResolver, ResolvedDataset from pt_snap_cli.core.errors import ( DatabaseMissingError, DatabaseSchemaError, @@ -202,6 +204,7 @@ def execute_query( *, exact_total: bool = False, timeout_s: float | None = None, + slice_index: int | None = None, ) -> QueryResult: resolved = self._focus_service.resolve_focus( explicit_db_path=db_path, @@ -211,10 +214,22 @@ def execute_query( if resolved.db_path is None: raise FocusNotConfiguredError("No database path specified and no database configured.") - ctx = self._validated_context(resolved.db_path) - target_device = self._resolve_device_id(ctx, resolved.device_id, device_id) - executor = self._get_executor(ctx) + dataset = DatasetResolver().inspect(resolved.db_path) + scope: dict[str, object] | None = None query_params = params or {} + if dataset is not None: + selected = dataset.device(device_id if device_id is not None else resolved.device_id) + member, query_params, scope = self._dataset_query( + dataset, selected.device_id, template, query_params, slice_index + ) + ctx = self._validated_context(member, generation=dataset.fingerprint) + target_device = selected.device_id + else: + if slice_index is not None: + raise InvalidParameterError("slice_index requires a manifest dataset.") + ctx = self._validated_context(resolved.db_path) + target_device = self._resolve_device_id(ctx, resolved.device_id, device_id) + executor = self._get_executor(ctx) timeout_s = resolve_query_timeout(timeout_s) started = time.monotonic() @@ -271,6 +286,86 @@ def execute_query( truncated=truncated, total_is_exact=total_is_exact, timeout_s=timeout_s, + scope=scope, + ) + + def _dataset_query( + self, + dataset: ResolvedDataset, + device_id: int, + template: str, + params: dict[str, Any], + slice_index: int | None, + ) -> tuple[Path, dict[str, Any], dict[str, object]]: + template_obj = get_query(template) + if template_obj is None: + raise TemplateNotFoundError(f"Template '{template}' not found") + if template != "event": + raise QueryExecutionError( + "Dataset queries currently support only event addressing; " + "cross-slice aggregation/lifecycle queries are not supported." + ) + try: + validated = template_obj.validate_params(params) + except (TypeError, ValueError) as exc: + raise TemplateRenderError(str(exc)) from exc + device = dataset.device(device_id) + if slice_index is not None: + paths = dataset.paths(QueryScope("slice", device_id, slice_index)) + item = device.slices[slice_index] + low, high = item.start_event_id, item.end_event_id + else: + low, high = 0, device.event_count - 1 + event_id = validated.get("id") + start = validated.get("min_id") + end = validated.get("max_id") + if event_id is not None: + if ( + not isinstance(event_id, int) + or isinstance(event_id, bool) + or not low <= event_id <= high + ): + raise InvalidParameterError("Event ID must be a real ID within the selected scope.") + low = high = event_id + else: + for value in (start, end): + if value is not None and ( + not isinstance(value, int) + or isinstance(value, bool) + or not low <= value <= high + ): + raise InvalidParameterError( + "Event range must lie within the selected real scope." + ) + low = start if isinstance(start, int) else low + high = end if isinstance(end, int) else high + paths = dataset.paths( + QueryScope("event_range", device_id, start_event_id=low, end_event_id=high) + ) + if len(paths) != 1: + raise QueryExecutionError( + "Event range spans slices; cross-slice execution is not supported. " + "Select a real id, a single-slice range, or --slice." + ) + item = next(item for item in device.slices if dataset.root / item.file == paths[0]) + return ( + paths[0], + { + **params, + "min_id": max(low, start) if isinstance(start, int) else low, + "max_id": min(high, end) if isinstance(end, int) else high, + }, + { + "kind": "event_range", + "device_id": device_id, + "slice_index": item.index, + "first_event_id": low, + "last_event_id": high, + "db_path": str(paths[0]), + "dataset_path": str(dataset.root), + "fingerprint": dataset.fingerprint, + "boundary_events_included": False, + }, ) def _get_executor(self, ctx: Context) -> QueryExecutor: @@ -308,8 +403,10 @@ def _resolve_device_id( raise InvalidDeviceError("No devices found in database.") return ctx.device_ids[0] - def _validated_context(self, db_path: Path) -> Context: + def _validated_context(self, db_path: Path, *, generation: str | None = None) -> Context: try: + if generation is not None: + return self._context_cache.get(db_path, generation=generation) return self._context_cache.get(db_path) except DatabaseNotFoundError as exc: raise DatabaseMissingError(str(exc)) from exc diff --git a/src/pt_snap_cli/snapshot/PROVENANCE.md b/src/pt_snap_cli/snapshot/PROVENANCE.md index f7d5edd..a84aa16 100644 --- a/src/pt_snap_cli/snapshot/PROVENANCE.md +++ b/src/pt_snap_cli/snapshot/PROVENANCE.md @@ -253,3 +253,36 @@ constructor/commit failure while preserving the original single-DB data contract Synthetic failure/equivalence tests and bilingual docs are locally authored; reviewed fixtures are unchanged. Existing source mappings and MIT relicensing records are retained. GUI interoperability and streaming/RSS claims are not made. + +## 2026-10-05 — Read-only dataset focus and addressing (issue #200) + +Local original reader changes in `core/dataset_resolver.py`, focus/overview/query +services and CLI/API adapters reuse the committed P0 compatibility-v1 validator. +Manifest/device/slice fields, real-event range routing and completion facts were +independently read via a fixed GitHub GET at Ascend/msinsight revision +`101f65b877a267ffd5f66ea3834706057ba243e5`, source +[manifest/routing interfaces](https://api.github.com/repos/Ascend/msinsight/git/blobs/8746d17db5a8e66b2ed0843962593c12851414e0). +The source carries Copyright (c) 2026 Huawei Technologies Co.,Ltd. and Mulan PSL +v2. No C++ implementation was copied; Python logic, synthetic SQLite regressions +and bilingual reader documentation are locally authored. Complete-readiness and +canonical artifact rules remain stricter than upstream building inspection. + +No allocator, importer, compatibility exporter, native-v2 staging protocol, +fixture, license or retained source mapping is changed. The reader reports text +stacks without promoting structured frames, refuses silent cross-slice template +execution, and never loads source pickle or repairs external databases. Context +LRU generation uses manifest/member content identity; real overview bounds exclude +negative boundary rows. No GUI, wheel/profile or performance acceptance is claimed. + +## 2026-10-05 — Dataset read-only WAL preflight correction (issue #200) + +Local original correction in `core/dataset_resolver.py`: verify every declared +member's canonical non-symlink path, WAL/journal/SHM sidecars (including dangling +aliases) and SQLite file-header read/write versions before any SQLite connection. +Checkpointed persistent-WAL databases are rejected even without existing sidecars, +because `mode=ro` can recreate WAL/SHM files. Synthetic real-SQLite regressions +record unchanged member/manifest hashes, inventory and no connection opens on +rejection across resolver, focus, CLI and API. No repair, checkpoint, journal-mode +change, artifact deletion or standalone-DB policy change is introduced. This is +not concurrent-producer locking. Existing attribution, source mappings, fixtures +and license records are unchanged; no live upstream artifact acceptance is claimed. diff --git a/tests/test_contract_cli_api.py b/tests/test_contract_cli_api.py index 9149593..4b93bdd 100644 --- a/tests/test_contract_cli_api.py +++ b/tests/test_contract_cli_api.py @@ -26,6 +26,7 @@ from pt_snap_cli.core.json_codec import JSON_SCHEMA_VERSION from pt_snap_cli.query.config import QueryParameter, QueryTemplate from pt_snap_cli.query.registry import QueryRegistry, register_query +from tests.test_dataset_focus import make_dataset runner = CliRunner() @@ -787,6 +788,68 @@ def test_capabilities_template_matches_template_info_json( assert "empty strings, NULL text or IDs" in " ".join(callstack["interpretation_limits"]) +@pytest.mark.parametrize("use_manifest", [False, True]) +def test_dataset_focus_overview_query_contract(tmp_path: Path, use_manifest: bool) -> None: + root = make_dataset(tmp_path / "dataset") + path = root / "manifest.json" if use_manifest else root + with SnapshotAnalyzer() as analyzer: + state = analyzer.set_focus(str(path), 2) + focus = _json_stdout(runner.invoke(app, ["focus", str(path), "--device", "2", "--json"])) + assert {key: focus[key] for key in _focus_payload(state)} == _focus_payload(state) + overview = _json_stdout(runner.invoke(app, ["overview", str(path), "--json"])) + api_overview = analyzer.get_database_overview() + assert {key: overview[key] for key in api_overview} == api_overview + metadata_result = runner.invoke(app, ["metadata", str(path), "--json"]) + assert metadata_result.exit_code == 0, metadata_result.output + assert json.loads(metadata_result.stdout) == analyzer.get_database_metadata() + for params, selector in [ + ({"id": 4}, []), + ({}, ["--slice", "1"]), + ({"min_id": 2, "max_id": 3}, []), + ]: + cli = _json_stdout( + runner.invoke( + app, + [ + "query", + str(path), + "--device", + "2", + "--template-use", + "event", + "--params", + json.dumps(params), + "--json", + *selector, + ], + ) + ) + api = analyzer.execute_query( + "event", params, device_id=2, slice_index=1 if selector else None + ) + assert {key: cli[key] for key in api} == api + assert cli["scope"]["boundary_events_included"] is False + error = runner.invoke( + app, ["query", str(path), "--template-use", "event", "--params", '{"id": -1}', "--json"] + ) + assert error.exit_code == 1 + assert json.loads(error.stderr)["error"]["code"] == "INVALID_PARAMETER" + failure = runner.invoke( + app, ["query", str(path), "--template-use", "memory_peak", "--json"] + ) + assert failure.exit_code == 1 + assert "cross-slice" in json.loads(failure.stderr)["error"]["message"] + manifest = root / "manifest.json" + data = json.loads(manifest.read_text()) + data["status"] = "building" + manifest.write_text(json.dumps(data)) + error = runner.invoke(app, ["focus", str(path), "--json"]) + assert error.exit_code == 1 + assert json.loads(error.stderr)["error"]["code"] == "DATABASE_SCHEMA_INVALID" + with pytest.raises(ValueError, match="complete"): + analyzer.set_focus(str(path)) + + def test_overview_error_contract_maps_four_domain_failures(tmp_path: Path) -> None: missing = runner.invoke(app, ["overview", str(tmp_path / "missing.db"), "--json"]) missing_error = json.loads(missing.stderr)["error"] diff --git a/tests/test_dataset_focus.py b/tests/test_dataset_focus.py new file mode 100644 index 0000000..3c5a489 --- /dev/null +++ b/tests/test_dataset_focus.py @@ -0,0 +1,483 @@ +"""Synthetic finalized artifacts: no pickle, metadata or optional reference tables.""" + +import hashlib +import json +import os +import sqlite3 +from contextlib import closing +from pathlib import Path + +import pytest +from typer.testing import CliRunner + +from pt_snap_cli.api import SnapshotAnalyzer +from pt_snap_cli.cli import app +from pt_snap_cli.completion import complete_device_ids +from pt_snap_cli.core.context_cache import ContextCache +from pt_snap_cli.core.dataset_contract import ACTION_NAMES, BLOCK_STATES, QueryScope +from pt_snap_cli.core.dataset_resolver import DatasetResolver +from pt_snap_cli.core.errors import ( + DatabaseSchemaError, + InvalidDeviceError, + InvalidParameterError, + QueryExecutionError, + TemplateNotFoundError, + TemplateRenderError, +) +from pt_snap_cli.core.focus_service import FocusService + +runner = CliRunner() + + +@pytest.fixture(autouse=True) +def isolate(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + monkeypatch.setenv("HOME", str(tmp_path)) + monkeypatch.delenv("PT_SNAP_DB_PATH", raising=False) + monkeypatch.setattr(Path, "home", lambda: tmp_path) + + +def make_dataset(root, devices=(0, 2), slices=3): + root.mkdir() + data = { + "schemaVersion": 1, + "status": "complete", + "sourceFile": "/archived/missing.pkl", + "cacheHash": "", + "eventsPerSlice": 2, + "devices": {}, + } + for device in devices: + directory = root / f"device_{device}" + directory.mkdir() + records = [] + for index in range(slices): + file = f"device_{device}/slice_{index:05d}.db" + records.append( + { + "index": index, + "startEventId": index * 2, + "endEventId": index * 2 + 1, + "file": file, + "ready": True, + } + ) + with closing(sqlite3.connect(root / file)) as conn, conn: + conn.executescript( + 'CREATE TABLE dictionary ("table" TEXT,"column" TEXT,"key" TEXT,"value" TEXT);' + f"CREATE TABLE trace_entry_{device} (id INTEGER PRIMARY KEY, action INTEGER," + "address INTEGER,size INTEGER,stream INTEGER,allocated INTEGER,active INTEGER," + "reserved INTEGER,callstack TEXT);" + f"CREATE TABLE block_{device} (id INTEGER PRIMARY KEY,address INTEGER,size INTEGER," + "requestedSize INTEGER,state INTEGER,allocEventId INTEGER,freeEventId INTEGER);" + ) + conn.executemany( + "INSERT INTO dictionary VALUES (?,?,?,?)", + [ + (f"trace_entry_{device}", "action", str(key), value) + for key, value in enumerate(ACTION_NAMES) + ] + + [ + (f"block_{device}", "state", str(key), value) for key, value in BLOCK_STATES + ], + ) + conn.executemany( + f"INSERT INTO trace_entry_{device} VALUES (?,4,100,8,0,8,8,16,?)", + [ + (event, f"device{device}:event{event}") + for event in range(index * 2, index * 2 + 2) + ], + ) + conn.execute( + f"INSERT INTO trace_entry_{device} VALUES (-1,2,100,16,0,999,999,999,'boundary')" + ) + data["devices"][str(device)] = { + "eventCount": slices * 2, + "sliceCount": slices, + "readySlices": list(range(slices)), + "slices": records, + } + (root / "manifest.json").write_text(json.dumps(data), encoding="utf-8") + return root + + +def hashes(root): + return { + str(p.relative_to(root)): hashlib.sha256(p.read_bytes()).hexdigest() + for p in root.rglob("*") + if p.is_file() + } + + +@pytest.mark.parametrize("manifest_path", [False, True]) +def test_dataset_focus_readonly_and_complete_addressing(tmp_path, manifest_path): + root = make_dataset(tmp_path / "capture.msinsight") + path = root / "manifest.json" if manifest_path else root + before = hashes(root) + with SnapshotAnalyzer() as analyzer: + state = analyzer.set_focus(str(path), device_id=2) + assert state.available_devices == [0, 2] + assert state.callstack_layout == "v1" + assert not (tmp_path / ".pt-snap/focus.json").exists() + overview = analyzer.get_database_overview() + assert overview["devices"] == [ + {"device_id": d, "first_event_id": 0, "last_event_id": 5} for d in (0, 2) + ] + assert overview["dataset"]["format"] == "compatibility-v1" + assert overview["dataset"]["structured_frames"] is False + assert overview["dataset"]["text_callstacks"] is True + assert overview["dataset"]["cross_slice_queries"] is False + assert overview["dataset"]["boundary_event_count"] == 6 + assert analyzer.get_database_metadata()["status"] == "unavailable" + for device in (0, 2): + for event in range(6): + result = analyzer.execute_query("event", {"id": event}, device_id=device) + assert result["rows"][0]["callstack"] == f"device{device}:event{event}" + assert result["scope"]["slice_index"] == event // 2 + assert len(analyzer.context_cache) <= 4 + dataset = DatasetResolver().inspect(path) + assert dataset is not None + assert len(dataset.paths(QueryScope("dataset"))) == 6 + assert len(dataset.paths(QueryScope("device", 2))) == 3 + assert len(dataset.paths(QueryScope("event_range", 2, start_event_id=1, end_event_id=4))) == 3 + assert before == hashes(root) + assert not list(root.rglob("*-journal")) + assert not list(root.rglob("*-wal")) + + +def test_precedence_device_and_completion(tmp_path, monkeypatch): + root = make_dataset(tmp_path / "dataset") + service = FocusService() + service.set_global_focus(root, 0) + service.set_project_focus(root / "manifest.json", 2) + with SnapshotAnalyzer() as analyzer: + assert analyzer.execute_query("event", {"id": 4})["device_id"] == 2 + assert analyzer.execute_query("event", {"id": 4}, device_id=0)["device_id"] == 0 + assert complete_device_ids() == ["0", "2"] + monkeypatch.setenv("PT_SNAP_DB_PATH", str(root)) + with SnapshotAnalyzer() as analyzer: + assert analyzer.execute_query("event", {"id": 4})["device_id"] == 0 + with SnapshotAnalyzer(root / "manifest.json", 2) as analyzer: + assert analyzer.execute_query("event", {"id": 4})["device_id"] == 2 + before = json.loads((tmp_path / ".pt-snap/focus.json").read_text()) + monkeypatch.delenv("PT_SNAP_DB_PATH") + state = service.set_device(0) + assert state.available_devices == [0, 2] + assert set(json.loads((tmp_path / ".pt-snap/focus.json").read_text())) == { + "db_path", + "device_id", + } + assert set(before) == {"db_path", "device_id"} + + +@pytest.mark.parametrize( + "template,params,slice_index,error", + [ + ("event", {}, None, QueryExecutionError), + ("event", {"min_id": 1, "max_id": 2}, None, QueryExecutionError), + ("event", {"id": -1}, None, InvalidParameterError), + ("event", {"id": 6}, None, InvalidParameterError), + ("event", {"id": 4}, 0, InvalidParameterError), + ("event", {"min_id": -1}, 0, InvalidParameterError), + ("event", {"min_id": 3, "max_id": 2}, None, InvalidParameterError), + ("event", {}, 99, InvalidParameterError), + ("event", {}, True, InvalidParameterError), + ("event", {"unknown": 0}, 0, TemplateRenderError), + ("memory_peak", {}, None, QueryExecutionError), + ("leak_detection", {}, 0, QueryExecutionError), + ("missing_template", {}, None, TemplateNotFoundError), + ], +) +def test_unsupported_scope_errors(tmp_path, template, params, slice_index, error): + root = make_dataset(tmp_path / "dataset") + with SnapshotAnalyzer(root) as analyzer, pytest.raises(error): + analyzer.execute_query(template, params, slice_index=slice_index) + + +def test_slice_and_single_slice_range_exclude_boundary_and_preserve_filters(tmp_path): + root = make_dataset(tmp_path / "dataset") + with SnapshotAnalyzer(root) as analyzer: + result = analyzer.execute_query("event", slice_index=1, exact_total=True) + assert [row["id"] for row in result["rows"]] == [2, 3] + assert result["total"] == 2 and result["total_is_exact"] + assert result["scope"]["boundary_events_included"] is False + assert analyzer.execute_query("event", {"min_id": 2, "max_id": 3})["rows"] == result["rows"] + assert analyzer.execute_query("event", {"id": 2, "min_id": 3})["rows"] == [] + with pytest.raises(InvalidDeviceError): + analyzer.execute_query("event", {"id": 0}, device_id=99) + with pytest.raises(InvalidDeviceError): + analyzer.set_focus(str(root), device_id=99) + + +@pytest.mark.parametrize( + "failure", + ["building", "version", "missing", "duplicate", "symlink", "wal", "corrupt", "readiness"], +) +def test_invalid_dataset_is_not_partial_focus(tmp_path, failure): + root = make_dataset(tmp_path / "dataset") + manifest = root / "manifest.json" + data = json.loads(manifest.read_text()) + if failure in ("building", "version", "readiness"): + if failure == "building": + data["status"] = "building" + elif failure == "version": + data["schemaVersion"] = 999 + else: + data["devices"]["0"]["slices"][0]["ready"] = False + manifest.write_text(json.dumps(data)) + elif failure == "duplicate": + manifest.write_text( + manifest.read_text().replace( + '"schemaVersion": 1', '"schemaVersion": 1, "schemaVersion": 1' + ) + ) + elif failure == "symlink": + # Keep original data in place; replace the directory input with an alias. + alias = tmp_path / "alias" + alias.symlink_to(root, target_is_directory=True) + root = alias + elif failure == "wal": + (root / "device_0/slice_00000.db-wal").touch() + elif failure == "corrupt": + (root / "device_0/slice_00000.db").write_bytes(b"not SQLite") + else: + (root / "device_0/slice_00000.db").unlink() + with SnapshotAnalyzer() as analyzer: + with pytest.raises(ValueError): + analyzer.set_focus(str(root)) + assert analyzer.get_focus().db_path is None + with pytest.raises(ValueError): + analyzer.get_database_overview(str(root)) + with pytest.raises(DatabaseSchemaError): + FocusService().set_project_focus(root) + assert not (tmp_path / ".pt-snap/focus.json").exists() + + +def test_content_generation_reopens_cached_context_and_borrowed_close(tmp_path): + root = make_dataset(tmp_path / "dataset") + cache = ContextCache(maxsize=1) + try: + with SnapshotAnalyzer(root, context_cache=cache) as analyzer: + analyzer.execute_query("event", {"id": 0}) + path = root / "device_0/slice_00000.db" + old = cache.get(path, generation=DatasetResolver().inspect(root).fingerprint) + stamp = path.stat() + with closing(sqlite3.connect(path)) as conn, conn: + conn.execute("UPDATE trace_entry_0 SET callstack='modified:evt0' WHERE id=0") + os.utime(path, ns=(stamp.st_atime_ns, stamp.st_mtime_ns)) + assert ( + analyzer.execute_query("event", {"id": 0})["rows"][0]["callstack"] + == "modified:evt0" + ) + assert old._conn is None + manifest = root / "manifest.json" + data = json.loads(manifest.read_text()) + data["cacheHash"] = "new-generation" + manifest.write_text(json.dumps(data)) + analyzer.execute_query("event", {"id": 0}) + analyzer.execute_query("event", {"id": 4}) + assert len(cache) == 1 + assert len(cache) == 1 # Injected cache still caller-owned. + with pytest.raises(RuntimeError, match="closed"): + analyzer.get_focus() + finally: + cache.close() + assert len(cache) == 0 + + +def test_single_db_native_and_inline_keep_original_query_behavior(tmp_path): + root = make_dataset(tmp_path / "dataset") + path = root / "device_0/slice_00000.db" + for native in (False, True): + if native: + with closing(sqlite3.connect(path)) as conn, conn: + conn.execute("CREATE TABLE callstack (id INTEGER PRIMARY KEY, callstack TEXT)") + conn.execute("INSERT INTO callstack VALUES (0, 'native')") + conn.execute("ALTER TABLE trace_entry_0 DROP COLUMN callstack") + conn.execute("ALTER TABLE trace_entry_0 ADD COLUMN callstackId INTEGER") + conn.execute("UPDATE trace_entry_0 SET callstackId=0") + with SnapshotAnalyzer(path) as analyzer: + assert analyzer.get_focus().callstack_layout == ("v2" if native else "v1") + assert analyzer.execute_query("event", {"id": -1})["rows"][0]["id"] == -1 + assert analyzer.get_database_overview()["devices"][0]["first_event_id"] == 0 + with pytest.raises(InvalidParameterError): + analyzer.execute_query("event", slice_index=0) + if native: + with pytest.raises(DatabaseSchemaError): + DatasetResolver().inspect(root) # Never label native v2 as compatibility v1. + + +def test_no_manifest_private_staging_is_rejected(tmp_path): + root = tmp_path / "private-native-staging" + root.mkdir() + with pytest.raises(DatabaseSchemaError, match="manifest.json"): + DatasetResolver().inspect(root) + assert not list(root.iterdir()) + + +def test_single_slice_default_is_full_real_trace(tmp_path): + root = make_dataset(tmp_path / "dataset", slices=1) + with SnapshotAnalyzer(root) as analyzer: + assert [row["id"] for row in analyzer.execute_query("event")["rows"]] == [0, 1] + + +def test_cli_releases_owned_context_and_sqlite_is_readonly(tmp_path, monkeypatch): + root = make_dataset(tmp_path / "dataset") + opened = [] + original = ContextCache.get + + def get(cache, path, **kwargs): + context = original(cache, path, **kwargs) + opened.append(context) + with context.connect() as conn, pytest.raises(sqlite3.OperationalError, match="readonly"): + conn.execute("CREATE TABLE illegal (id INTEGER)") + return context + + monkeypatch.setattr(ContextCache, "get", get) + result = runner.invoke( + app, ["query", str(root), "--template-use", "event", "--params", '{"id": 0}', "--json"] + ) + assert result.exit_code == 0, result.output + assert opened and all(context._conn is None for context in opened) + failure = runner.invoke( + app, + [ + "query", + str(root), + "--template-use", + "event", + "--params", + '{"id": 0}', + "--timeout", + "not-a-number", + "--json", + ], + ) + assert failure.exit_code != 0 + assert all(context._conn is None for context in opened) + + +@pytest.mark.parametrize("suffix", ["-wal", "-journal", "-shm"]) +@pytest.mark.parametrize("dangling", [False, True]) +def test_live_sidecar_rejected_before_any_sqlite_open(tmp_path, monkeypatch, suffix, dangling): + root = make_dataset(tmp_path / "dataset") + # The last member must be checked before even the first slice is opened. + sidecar = root / f"device_2/slice_00002.db{suffix}" + if dangling: + sidecar.symlink_to(tmp_path / "missing-sidecar-target") + assert sidecar.is_symlink() and not sidecar.exists() + else: + sidecar.touch() + before = hashes(root) + inventory = sorted(str(p.relative_to(root)) for p in root.rglob("*")) + + def forbidden(*args, **kwargs): + raise AssertionError("SQLite opened before live sidecar rejection") + + monkeypatch.setattr(sqlite3, "connect", forbidden) + with pytest.raises(DatabaseSchemaError, match="live SQLite sidecar"): + DatasetResolver().inspect(root) + assert before == hashes(root) + assert inventory == sorted(str(p.relative_to(root)) for p in root.rglob("*")) + if dangling: + assert sidecar.readlink() == tmp_path / "missing-sidecar-target" + + +@pytest.mark.parametrize("manifest_path", [False, True]) +@pytest.mark.parametrize("operation", ["inspect", "focus", "api", "cli"]) +def test_checkpointed_wal_rejected_without_artifact_changes( + tmp_path, monkeypatch, manifest_path, operation +): + root = make_dataset(tmp_path / "dataset") + member = root / "device_2/slice_00002.db" + # A genuine checkpointed WAL database retains header versions 2/2 after + # the last writer closes, even though no WAL/SHM file remains on disk. + with closing(sqlite3.connect(member)) as conn: + assert conn.execute("PRAGMA journal_mode=WAL").fetchone() == ("wal",) + with member.open("rb") as source: + assert source.read(20)[18:20] == b"\x02\x02" + assert not list(root.rglob("*-wal")) and not list(root.rglob("*-shm")) + before = hashes(root) + inventory = sorted(str(p.relative_to(root)) for p in root.rglob("*")) + opened = [] + original = sqlite3.connect + + def connect(*args, **kwargs): + opened.append(args[0]) + return original(*args, **kwargs) + + monkeypatch.setattr(sqlite3, "connect", connect) + path = root / "manifest.json" if manifest_path else root + if operation == "inspect": + with pytest.raises(DatabaseSchemaError): + DatasetResolver().inspect(path) + elif operation == "focus": + with pytest.raises(DatabaseSchemaError): + FocusService().set_project_focus(path) + elif operation == "api": + with SnapshotAnalyzer() as analyzer: + with pytest.raises(ValueError): + analyzer.set_focus(str(path)) + assert analyzer.get_focus().db_path is None + else: + result = runner.invoke(app, ["focus", str(path), "--json"]) + assert result.exit_code == 1 + assert json.loads(result.stderr)["error"]["code"] == "DATABASE_SCHEMA_INVALID" + assert before == hashes(root) + assert inventory == sorted(str(p.relative_to(root)) for p in root.rglob("*")) + assert opened == [] + assert not (tmp_path / ".pt-snap/focus.json").exists() + + +@pytest.mark.parametrize("alias_parent", [False, True]) +def test_member_alias_rejected_before_header_read(tmp_path, monkeypatch, alias_parent): + source = make_dataset(tmp_path / "source", devices=(0,), slices=1) + root = tmp_path / "dataset" + root.mkdir() + (root / "manifest.json").write_bytes((source / "manifest.json").read_bytes()) + if alias_parent: + (root / "device_0").symlink_to(source / "device_0", target_is_directory=True) + else: + (root / "device_0").mkdir() + (root / "device_0/slice_00000.db").symlink_to(source / "device_0/slice_00000.db") + before = hashes(source) + original = Path.open + + def open_path(path, *args, **kwargs): + if path.suffix == ".db": + raise AssertionError("Member alias followed before header rejection") + return original(path, *args, **kwargs) + + monkeypatch.setattr(Path, "open", open_path) + with pytest.raises(DatabaseSchemaError, match="symlink"): + DatasetResolver().inspect(root) + monkeypatch.setattr(Path, "open", original) + assert before == hashes(source) + + +def test_manifest_change_during_validation_fails(tmp_path, monkeypatch): + import pt_snap_cli.core.dataset_resolver as module + + root = make_dataset(tmp_path / "dataset") + original = module.validate_dataset + + def validate(path): + result = original(path) + manifest = root / "manifest.json" + data = json.loads(manifest.read_text()) + data["cacheHash"] = "replaced" + manifest.write_text(json.dumps(data)) + return result + + monkeypatch.setattr(module, "validate_dataset", validate) + with pytest.raises(DatabaseSchemaError, match="changed during inspection"): + DatasetResolver().inspect(root) + + +def test_unknown_extension_never_promotes_frames(tmp_path): + root = make_dataset(tmp_path / "dataset") + path = root / "manifest.json" + data = json.loads(path.read_text()) + data["extensions"] = {"frames": {"version": 999}} + path.write_text(json.dumps(data)) + assert DatasetResolver().inspect(root).to_dict()["structured_frames"] is False