Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,10 @@ deterministic names, replay validation, and failure-safe publication.

See the [full quick start guide](docs/en/quickstart.md) for a walkthrough.

Complete compatibility-v1 sharded artifacts can be focused by directory or
`manifest.json`, without the archived pickle. See [dataset focus and addressing](docs/en/sharded-snapshotdb.md#complete-dataset-focus-and-addressing-p1)
for read-only overview, event routing, `--slice`, and explicit cross-slice limits.

## Commands

| Command | Description |
Expand Down
4 changes: 4 additions & 0 deletions README_zh.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,10 @@ pt-snap split snapshot.pkl --slices 4 --output snapshot-slices

完整的入门指南见 [Quick Start](docs/zh/quickstart.md)。

完整兼容 v1 分片产物可以目录或 `manifest.json` 为 focus,无需已归档 pickle。
只读 overview、事件路由、`--slice` 和跨片限制见
[完整数据集 focus 与寻址](docs/zh/sharded-snapshotdb.md#完整数据集-focus-与寻址p1)。

## 命令

| 命令 | 说明 |
Expand Down
80 changes: 75 additions & 5 deletions docs/en/sharded-snapshotdb.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,9 @@
This freezes an artifact contract and shared validation models, **not a new CLI
exporter or cross-slice query engine**. Existing `import` still defaults to one
native v2 DB; `split` still produces replayable pickle/JSON. Existing single-DB
v1/v2 analysis and focus are unchanged. The `complete` gate below applies only to
this new dataset contract, not to legacy single-DB analysis.
v1/v2 single-DB analysis is unchanged. The P1 reader below additionally accepts a
complete compatibility-v1 manifest or directory as focus. The `complete` gate
applies only to datasets, not to legacy single-DB analysis.

`pt_snap_cli.core.dataset_contract` provides `parse_manifest`,
`validate_event_ids`, `validate_dataset`, immutable manifest/device/slice models,
Expand Down Expand Up @@ -166,9 +167,9 @@ which P0 does not implement. Text stacks cannot substitute for these guarantees.
| `event_range` | `device_id`, inclusive `start_event_id/end_event_id` within the real trace, possibly across slices |

There is no global event order across devices. Boundary IDs cannot be range
endpoints. `query_execution=False` explicitly declares that cross-slice execution
is deferred; passing scope validation does not make existing CLI/API accept a
manifest or directory.
endpoints. P0 `DatasetValidation.query_execution=False` describes validation,
not execution. P1 accepts complete manifests/directories through the resolver
below; cross-slice execution remains deferred.

```python
from pt_snap_cli.core.dataset_contract import QueryScope, validate_dataset
Expand All @@ -180,6 +181,75 @@ QueryScope("event_range", device_id=0, start_event_id=1, end_event_id=3).validat
assert not inspection.structured_frames
```

## Complete dataset focus and addressing (P1)

A relocated compatibility-v1 directory or its `manifest.json` can be focused
without the original pickle, import metadata, or reference/extension tables:

```bash
pt-snap focus /capture/snapshot.pkl.msinsight --device 0 --json
pt-snap overview --json
pt-snap query --template-use event --params '{"id": 3}' --json
pt-snap query --template-use event --slice 1 --json
```

`overview.dataset` reports format, manifest version/status, content fingerprint,
real/boundary counts, per-device slice paths/ranges and capabilities. Event bounds
exclude negative synthetic rows (also for standalone DB overview). Text stacks
are available, but not necessarily populated; structured frames and cross-slice
queries are **unavailable**, including when unknown extensions claim them.
Dataset-wide import metadata is unavailable; per-slice metadata is not a source
attestation for the entire dataset.

Path precedence stays explicit → `PT_SNAP_DB_PATH` → nearest project focus → legacy
global config; device precedence stays explicit → focused → first discovered.
CLI focus only writes the chosen project/global focus file, never the artifact or
detected layout. API session focus does not write it:

```python
from pathlib import Path
from pt_snap_cli import SnapshotAnalyzer

with SnapshotAnalyzer(Path("/capture/snapshot.pkl.msinsight"), device_id=0) as analyzer:
event = analyzer.execute_query("event", {"id": 3})
page = analyzer.execute_query("event", slice_index=1, exact_total=True)
print(event["scope"], page["total_is_exact"])
```

Only the `event` template is currently supported through dataset focus: a real ID
routes to its containing slice; an inclusive `min_id/max_id` range must fit one
slice; `--slice`/`slice_index` lists that slice's real events. Negative IDs are not
whole-dataset selectors. Output `scope` identifies the actual DB/device/slice,
restricted real interval, dataset fingerprint and boundary exclusion. Totals and
pagination apply to this scope, **not** to an implicitly truncated full dataset.
Caller parameter defaults in CLI `effective_params` are additionally constrained
by `scope`. Cross-slice ranges, unbounded multi-slice queries and aggregate or
lifecycle templates fail explicitly; they never silently choose the first/latest
slice. Direct single-DB v1/v2 queries retain their existing semantics.

`core.dataset_resolver.DatasetResolver.inspect(path)` returns an immutable
`ResolvedDataset` (or `None` for a standalone DB). Its `paths(QueryScope(...))`
addresses all matching files even for a cross-slice range, but does not execute it.
It revalidates the finalized P0 contract on each call and hashes the manifest and
members; no unbounded manifest/connection cache is retained. Hashing uses bounded
chunks and is proportional to artifact size, not a performance claim. Context LRU
size defaults to four; dataset generation changes invalidate contexts on next
lookup even when size/mtime are unchanged. Analyzer-owned caches close on exit;
injected caches remain caller-owned. Validation opens/closes slices sequentially.

Building, missing/inconsistent slices, unknown base versions and unsafe paths
are rejected before focus writes or query execution. Before opening **any** slice,
the resolver checks every member's canonical non-symlink path, SQLite header and
WAL/journal/SHM sidecars (including dangling symlinks). Persistent WAL mode is
rejected even after checkpointing has removed its sidecars: `mode=ro` can recreate
them. The reader does not repair, checkpoint, delete sidecars or change journal
mode; standalone DB behavior is unchanged. This is read-only inspection, not a
filesystem sandbox or a concurrent producer lock. Native-v2 replay's private staging directory has **no published manifest**
and is rejected as a dataset; its individual DBs remain readable via the existing
single-DB entry. A future compatibility-v1 producer can use this same reader
without producer-specific metadata. No native-v2 manifest protocol, exporter,
cache publication, migration, or cross-slice lifecycle reconstruction is invented.

## Internal continuous replay (P1)

`core.sharded_replay_service.ShardedReplayService.stage(source, directory,
Expand Down
62 changes: 59 additions & 3 deletions docs/zh/sharded-snapshotdb.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,8 @@

本阶段冻结产物协议和共享校验模型,**不新增 CLI 导出器或跨片查询引擎**。现有
`import` 默认仍生成一个原生 v2 DB;`split` 仍输出可回放 pickle/JSON。现有单库
v1/v2 分析与 focus 不变。下述 `complete` 门槛只约束新数据集合同,不约束旧单库分析。
v1/v2 单库分析不变。下述 P1 reader 新增将 complete 兼容 v1 manifest 或目录作为
focus;`complete` 门槛只约束数据集,不约束旧单库分析。

`pt_snap_cli.core.dataset_contract` 提供 `parse_manifest`、`validate_event_ids`、
`validate_dataset`、不可变 manifest/device/slice 模型、`QueryScope` 和带
Expand Down Expand Up @@ -135,8 +136,9 @@ dictionary 的十进制整数 key 以文本存储,使用真实带设备后缀
| `slice` | `device_id`, `slice_index`;该片真实区间,边界独立 |
| `event_range` | `device_id`、设备内闭区间 `start_event_id/end_event_id`;可跨片 |

设备间没有全局事件顺序。边界 ID 不能作为范围端点。`query_execution=False`
明确表示跨片执行留后续;scope 校验成功不使现有 CLI/API 自动接受 manifest 或目录。
设备间没有全局事件顺序。边界 ID 不能作为范围端点。P0 的
`DatasetValidation.query_execution=False` 描述校验而非执行。下述 P1 resolver
使 CLI/API 接受完整 manifest/目录;跨片执行仍留后续。

```python
from pt_snap_cli.core.dataset_contract import QueryScope, validate_dataset
Expand All @@ -148,6 +150,60 @@ QueryScope("event_range", device_id=0, start_event_id=1, end_event_id=3).validat
assert not inspection.structured_frames
```

## 完整数据集 focus 与寻址(P1)

已搬迁的兼容 v1 目录或其 `manifest.json` 可直接 focus;不要求原始 pickle、导入
metadata、辅助引用表或扩展表:

```bash
pt-snap focus /capture/snapshot.pkl.msinsight --device 0 --json
pt-snap overview --json
pt-snap query --template-use event --params '{"id": 3}' --json
pt-snap query --template-use event --slice 1 --json
```

`overview.dataset` 输出格式、manifest 版本/状态、内容指纹、真实/边界事件数、逐设备
分片路径/范围及能力。事件范围排除负 ID 合成行(单库 overview 也如此)。支持文本调用栈,
不保证每行有栈;结构化 frames 与跨片查询**不可用**,未知扩展的能力声明不会提升它们。
数据集级导入 metadata 为 unavailable;逐片 metadata 不是整份数据集的来源证明。

路径优先级仍为显式 → `PT_SNAP_DB_PATH` → 最近项目 focus → legacy global config;
设备优先级仍为显式 → focused → 首个发现设备。CLI focus 只写选定的项目/全局 focus 文件,
不写产物或检测布局;API 会话 focus 不写配置:

```python
from pathlib import Path
from pt_snap_cli import SnapshotAnalyzer

with SnapshotAnalyzer(Path("/capture/snapshot.pkl.msinsight"), device_id=0) as analyzer:
event = analyzer.execute_query("event", {"id": 3})
page = analyzer.execute_query("event", slice_index=1, exact_total=True)
print(event["scope"], page["total_is_exact"])
```

数据集 focus 当前只支持 `event` 模板:真实 ID 路由至所在片,闭区间 `min_id/max_id`
必须落在一个片内,`--slice`/`slice_index` 列出该片真实事件。负 ID 不能作为整份数据集
selector。输出 `scope` 标识实际 DB/device/slice、受限真实区间、数据集指纹和边界排除。
总数和分页针对该 scope,**不是**隐式截断的整份数据集。CLI `effective_params` 的调用者
参数默认值另受 `scope` 约束。跨片区间、无界多片查询、聚合/生命周期模板明确失败,不会
静默选首片/最新片。直接对单库 v1/v2 查询保持原有语义。

`core.dataset_resolver.DatasetResolver.inspect(path)` 返回不可变 `ResolvedDataset`
(单库返回 `None`);其 `paths(QueryScope(...))` 可寻址跨片范围的全部文件,但不执行查询。
每次调用重新校验最终化 P0 合同并对 manifest/所有成员取内容哈希,不保存无界 manifest/
连接缓存。哈希按有界块读取,工作量与产物大小成正比,不宣称性能收益。Context LRU 默认
最多四个;数据集 generation 变化在下次 lookup 使旧 Context 失效,即使 size/mtime 相同。
analyzer 自有缓存在退出时关闭,注入缓存仍由调用者所有;校验逐片打开并关闭连接。

building、缺片/矛盾片、未知基础版本、不安全路径在 focus 写入或查询前拒绝。在打开
**任何**分片前,resolver 先检查全部成员的规范非符号链接路径、SQLite header 及
WAL/journal/SHM sidecar(包含悬空符号链接)。即使 checkpoint 已移除 sidecar,持久 WAL
模式仍被拒绝:`mode=ro` 也可能重新创建它们。reader 不修复、不执行 checkpoint、不删除
sidecar、不改变 journal mode;单库行为不变。这是只读检查,不是文件系统沙箱或并发生产端锁。native-v2 replay 的私有
staging **没有已发布 manifest**,不能作为数据集;其中单个 DB 仍由已有单库入口读取。
后续兼容 v1 生产端可直接使用同一 reader,不依赖生产端专属 metadata。本项不虚构
native-v2 manifest 协议,不新增导出、缓存发布、迁移或跨片生命周期重建。

## 内部持续回放(P1)

`core.sharded_replay_service.ShardedReplayService.stage(source, directory,
Expand Down
13 changes: 11 additions & 2 deletions src/pt_snap_cli/api.py
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,11 @@ class TemplateSummaryPayload(TypedDict):
category: str | None


class QueryResultPayload(TypedDict):
class _QueryScopePayload(TypedDict, total=False):
scope: dict[str, object]


class QueryResultPayload(_QueryScopePayload):
total: int
returned: int
device_id: int | None
Expand Down Expand Up @@ -209,6 +213,7 @@ def execute_query(
*,
exact_total: bool = False,
timeout_s: float | None = None,
slice_index: int | None = None,
) -> QueryResultPayload:
self._ensure_open()
try:
Expand All @@ -220,10 +225,11 @@ def execute_query(
max_rows=max_rows,
exact_total=exact_total,
timeout_s=timeout_s,
slice_index=slice_index,
)
except FocusNotConfiguredError as exc:
raise RuntimeError("No database configured. Call set_focus() first.") from exc
return {
payload: QueryResultPayload = {
"total": result.total,
"returned": result.returned,
"device_id": result.device_id,
Expand All @@ -235,6 +241,9 @@ def execute_query(
"total_is_exact": result.total_is_exact,
"timeout_s": result.timeout_s,
}
if result.scope is not None:
payload["scope"] = result.scope
return payload

def list_capabilities(self) -> dict[str, object]:
self._ensure_open()
Expand Down
39 changes: 34 additions & 5 deletions src/pt_snap_cli/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,7 @@
json_success,
)
from pt_snap_cli.core.error_codes import DATABASE_NOT_FOUND, ERROR, INVALID_PARAMETER
from pt_snap_cli.core.errors import InvalidParameterError
from pt_snap_cli.query.config import OUTPUT_COLUMN_OPTIONAL
from pt_snap_cli.query.executor import reported_sql_limit
from pt_snap_cli.query.registry import discover_categories, get_query
Expand Down Expand Up @@ -236,7 +237,12 @@ def main(

@app.command("focus")
def focus_database(
db_path: Annotated[Path | None, typer.Argument(help="Path to SQLite database file")] = None,
db_path: Annotated[
Path | None,
typer.Argument(
help="Path to SQLite database, complete manifest.json, or dataset directory"
),
] = None,
device: Annotated[
int | None,
typer.Option(
Expand All @@ -257,7 +263,7 @@ def focus_database(
if db_path is None and device is None:
try:
state = focus_service.get_focus()
except FocusFileInvalidError as e:
except (FocusFileInvalidError, DatabaseSchemaError) as e:
_error_from_exc(e)

if json_output:
Expand Down Expand Up @@ -297,6 +303,7 @@ def focus_database(
FocusFileInvalidError,
FocusNotConfiguredError,
DatabaseMissingError,
DatabaseSchemaError,
InvalidDeviceError,
) as e:
_error_from_exc(e)
Expand Down Expand Up @@ -454,7 +461,10 @@ def split_snapshot(
@app.command("metadata")
def show_database_metadata(
db_path: Annotated[
Path | None, typer.Argument(help="Path to database file (optional if configured)")
Path | None,
typer.Argument(
help="Path to database, manifest.json, or dataset directory (optional if configured)"
),
] = None,
json_output: Annotated[bool, _json_flag()] = False,
) -> None:
Expand Down Expand Up @@ -491,7 +501,10 @@ def show_database_metadata(
@app.command("query")
def query_database(
db_path: Annotated[
Path | None, typer.Argument(help="Path to database file (optional if configured)")
Path | None,
typer.Argument(
help="Path to database, manifest.json, or dataset directory (optional if configured)"
),
] = None,
template_use: Annotated[
str | None,
Expand All @@ -509,6 +522,10 @@ def query_database(
autocompletion=complete_device_ids,
),
] = None,
slice_index: Annotated[
int | None,
typer.Option("--slice", help="Explicit dataset slice index for event inspection"),
] = None,
list_templates: Annotated[
bool, typer.Option("--list", help="List all available query templates")
] = False,
Expand Down Expand Up @@ -702,6 +719,7 @@ def query_database(
max_rows=max_rows,
exact_total=exact_total,
timeout_s=timeout,
slice_index=slice_index,
)
if json_output:
resolved = focus_service.resolve_focus(
Expand All @@ -725,6 +743,7 @@ def query_database(
total_is_exact=result.total_is_exact,
timeout_s=result.timeout_s,
rows=result.rows,
**({"scope": result.scope} if result.scope is not None else {}),
)
)
return
Expand Down Expand Up @@ -779,6 +798,7 @@ def query_database(
except (
TemplateRenderError,
QueryExecutionError,
InvalidParameterError,
DatabaseSchemaError,
json.JSONDecodeError,
) as e:
Expand Down Expand Up @@ -860,7 +880,10 @@ def show_capabilities(
@app.command("overview")
def show_database_overview(
db_path: Annotated[
Path | None, typer.Argument(help="Path to database file (optional if configured)")
Path | None,
typer.Argument(
help="Path to database, manifest.json, or dataset directory (optional if configured)"
),
] = None,
json_output: Annotated[bool, _json_flag()] = False,
) -> None:
Expand All @@ -882,6 +905,12 @@ def show_database_overview(
_emit_json(json_success(**service.overview_to_dict(overview)))
return
_print_database_overview(overview)
if overview.dataset is not None:
typer.echo(f"Dataset: {overview.dataset['format']} ({overview.dataset['status']})")
typer.echo(f"Slices/devices: {overview.dataset['devices']}")
typer.echo(
"Text callstacks available; structured frames and cross-slice queries unavailable."
)


# Flag-only options used when a parse failure happens before the --json callback.
Expand Down
5 changes: 5 additions & 0 deletions src/pt_snap_cli/completion.py
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,11 @@ def complete_device_ids() -> list[str]:
return []

try:
from pt_snap_cli.core.dataset_resolver import DatasetResolver

dataset = DatasetResolver().inspect(db_path)
if dataset is not None:
return [str(device) for device in dataset.device_ids]
uri = f"file:{db_path}?mode=ro"
with closing(sqlite3.connect(uri, uri=True)) as conn:
cursor = conn.cursor()
Expand Down
Loading